|
|
|
@@ -0,0 +1,74 @@
|
|
|
|
|
---
|
|
|
|
|
title: "The first secret is the one you can't commit"
|
|
|
|
|
date: 2026-06-28
|
|
|
|
|
summary: "Secrets management has a bootstrap paradox: the credential that pulls every other secret can't itself live in git. Here's how a homelab fleet breaks the cycle — zero plaintext secrets in any repo, and a clean rule for which is the one exception."
|
|
|
|
|
tags: ["security", "gitops", "secrets", "kubernetes", "external-secrets"]
|
|
|
|
|
draft: false
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|

|
|
|
|
|
|
|
|
|
|
Every "we do GitOps properly" story has a chicken-and-egg problem hiding in the first paragraph, and most
|
|
|
|
|
posts quietly skip it.
|
|
|
|
|
|
|
|
|
|
The pitch is clean: secrets never live in git. Instead, the cluster runs the [External Secrets
|
|
|
|
|
Operator](https://external-secrets.io) (ESO), which reads from a real secret store — here, a self-hosted
|
|
|
|
|
[Infisical](https://infisical.com) — and materialises a Kubernetes `Secret` for each app. Your repo only ever
|
|
|
|
|
contains an `ExternalSecret` manifest: a **pointer** ("give `immich` the value at key `IMMICH_DB_PASSWORD`"),
|
|
|
|
|
never a value. Beautiful. Auditable. Diff-able.
|
|
|
|
|
|
|
|
|
|
Except — ESO has to authenticate to Infisical somehow. That's a credential. Where does *it* live?
|
|
|
|
|
|
|
|
|
|
## Secret zero
|
|
|
|
|
|
|
|
|
|
You can't store the bootstrap credential in git (that's the whole point), and you can't fetch it from the
|
|
|
|
|
secret store (you need it *to reach* the secret store). This is **secret zero**: the one credential that the
|
|
|
|
|
entire chain hangs off, that has to be injected from outside the GitOps loop.
|
|
|
|
|
|
|
|
|
|
So you treat it as exactly that — special, minimal, and out-of-band:
|
|
|
|
|
|
|
|
|
|
- It's a **machine identity** scoped to *read-only*, and scoped per cluster. cave's ESO can't read alfred's
|
|
|
|
|
secrets and vice-versa. A leak is blast-radius-limited to one cluster's read path.
|
|
|
|
|
- It's applied **once, by hand** (or by a sealed bootstrap step), never committed. Everything downstream of it
|
|
|
|
|
is declarative.
|
|
|
|
|
- After it lands, a single `ClusterSecretStore` object — which *is* in git, because it's just a pointer to the
|
|
|
|
|
store plus a reference to secret zero — wires the whole cluster up.
|
|
|
|
|
|
|
|
|
|
From there the dam breaks in the good way: every app's `ExternalSecret` resolves through that store, ESO keeps
|
|
|
|
|
the `Secret` in sync, and your repo stays a map of *names*, not values.
|
|
|
|
|
|
|
|
|
|
<!-- DIAGRAM: secret-zero bootstrap chain — out-of-band identity → ClusterSecretStore → ExternalSecrets → app Secrets -->
|
|
|
|
|

|
|
|
|
|
|
|
|
|
|
## The discipline around the one exception
|
|
|
|
|
|
|
|
|
|
One out-of-band credential is fine. The trap is letting it sprawl — or fumbling its rotation. Two rules earn
|
|
|
|
|
their keep:
|
|
|
|
|
|
|
|
|
|
**Per-cluster, least-privilege identities.** It's tempting to mint one powerful identity and reuse it
|
|
|
|
|
everywhere. Don't. A read-only identity per cluster means the bootstrap secret can *read* and nothing else, in
|
|
|
|
|
*one* place. The write path (provisioning new secrets) is a separate, rarely-used identity that doesn't sit on
|
|
|
|
|
every node.
|
|
|
|
|
|
|
|
|
|
**Never revoke a shared identity until every consumer has moved off it.** This sounds obvious and is the
|
|
|
|
|
single easiest way to take the fleet down. When you split a shared identity into per-cluster ones, the old one
|
|
|
|
|
stays valid until you've *verified* each cluster is happily authenticating on its own — then, and only then,
|
|
|
|
|
you revoke. Revoke-first-verify-later turns a tidy-up into an outage.
|
|
|
|
|
|
|
|
|
|
## What still belongs in git (encrypted)
|
|
|
|
|
|
|
|
|
|
A purist would stop here, but reality has a few things that genuinely need to live *in* the repo — a value a
|
|
|
|
|
bootstrap step reads before ESO is even running. For those, the answer isn't "commit it in plaintext and feel
|
|
|
|
|
bad," it's **SOPS + age**: the value is encrypted in git, decryptable only by a key that lives on the
|
|
|
|
|
operators' machines (and the cluster), never in the repo. Same principle as secret zero — the *decryption* key
|
|
|
|
|
is the out-of-band thing — applied to the handful of values that can't wait for the operator to spin up.
|
|
|
|
|
|
|
|
|
|
## The shape of it
|
|
|
|
|
|
|
|
|
|
The win isn't any one tool. It's the shape: **exactly one** credential lives outside GitOps, it's read-only
|
|
|
|
|
and per-cluster, and it's the seed the whole tree grows from. Everything else is a pointer you can show
|
|
|
|
|
your worst enemy. When someone asks "where are your secrets?", the honest answer is "in the store — the repo
|
|
|
|
|
just knows their *names*," and that one sentence is the whole security model.
|
|
|
|
|
|
|
|
|
|
*This runs across a five-cluster Talos homelab; the ESO + Infisical wiring, the per-cluster read-only
|
|
|
|
|
identities, and the revoke-order discipline are the load-bearing pieces.*
|