Add an optional hero field to the blog schema; render it on the post page and as the index card thumbnail. Ship 4 verified Nano Banana hero images (shipping/ init-gating/authentik/workload) as optimized webp (64-134KB). Publish the secret-zero post. (outbound hero pending — its file was a dup of shipping; secret-zero hero pending generation.)
This commit is contained in:
@@ -0,0 +1,74 @@
|
||||
---
|
||||
title: "The first secret is the one you can't commit"
|
||||
date: 2026-06-28
|
||||
summary: "Secrets management has a bootstrap paradox: the credential that pulls every other secret can't itself live in git. Here's how a homelab fleet breaks the cycle — zero plaintext secrets in any repo, and a clean rule for which is the one exception."
|
||||
tags: ["security", "gitops", "secrets", "kubernetes", "external-secrets"]
|
||||
draft: false
|
||||
---
|
||||
|
||||

|
||||
|
||||
Every "we do GitOps properly" story has a chicken-and-egg problem hiding in the first paragraph, and most
|
||||
posts quietly skip it.
|
||||
|
||||
The pitch is clean: secrets never live in git. Instead, the cluster runs the [External Secrets
|
||||
Operator](https://external-secrets.io) (ESO), which reads from a real secret store — here, a self-hosted
|
||||
[Infisical](https://infisical.com) — and materialises a Kubernetes `Secret` for each app. Your repo only ever
|
||||
contains an `ExternalSecret` manifest: a **pointer** ("give `immich` the value at key `IMMICH_DB_PASSWORD`"),
|
||||
never a value. Beautiful. Auditable. Diff-able.
|
||||
|
||||
Except — ESO has to authenticate to Infisical somehow. That's a credential. Where does *it* live?
|
||||
|
||||
## Secret zero
|
||||
|
||||
You can't store the bootstrap credential in git (that's the whole point), and you can't fetch it from the
|
||||
secret store (you need it *to reach* the secret store). This is **secret zero**: the one credential that the
|
||||
entire chain hangs off, that has to be injected from outside the GitOps loop.
|
||||
|
||||
So you treat it as exactly that — special, minimal, and out-of-band:
|
||||
|
||||
- It's a **machine identity** scoped to *read-only*, and scoped per cluster. cave's ESO can't read alfred's
|
||||
secrets and vice-versa. A leak is blast-radius-limited to one cluster's read path.
|
||||
- It's applied **once, by hand** (or by a sealed bootstrap step), never committed. Everything downstream of it
|
||||
is declarative.
|
||||
- After it lands, a single `ClusterSecretStore` object — which *is* in git, because it's just a pointer to the
|
||||
store plus a reference to secret zero — wires the whole cluster up.
|
||||
|
||||
From there the dam breaks in the good way: every app's `ExternalSecret` resolves through that store, ESO keeps
|
||||
the `Secret` in sync, and your repo stays a map of *names*, not values.
|
||||
|
||||
<!-- DIAGRAM: secret-zero bootstrap chain — out-of-band identity → ClusterSecretStore → ExternalSecrets → app Secrets -->
|
||||

|
||||
|
||||
## The discipline around the one exception
|
||||
|
||||
One out-of-band credential is fine. The trap is letting it sprawl — or fumbling its rotation. Two rules earn
|
||||
their keep:
|
||||
|
||||
**Per-cluster, least-privilege identities.** It's tempting to mint one powerful identity and reuse it
|
||||
everywhere. Don't. A read-only identity per cluster means the bootstrap secret can *read* and nothing else, in
|
||||
*one* place. The write path (provisioning new secrets) is a separate, rarely-used identity that doesn't sit on
|
||||
every node.
|
||||
|
||||
**Never revoke a shared identity until every consumer has moved off it.** This sounds obvious and is the
|
||||
single easiest way to take the fleet down. When you split a shared identity into per-cluster ones, the old one
|
||||
stays valid until you've *verified* each cluster is happily authenticating on its own — then, and only then,
|
||||
you revoke. Revoke-first-verify-later turns a tidy-up into an outage.
|
||||
|
||||
## What still belongs in git (encrypted)
|
||||
|
||||
A purist would stop here, but reality has a few things that genuinely need to live *in* the repo — a value a
|
||||
bootstrap step reads before ESO is even running. For those, the answer isn't "commit it in plaintext and feel
|
||||
bad," it's **SOPS + age**: the value is encrypted in git, decryptable only by a key that lives on the
|
||||
operators' machines (and the cluster), never in the repo. Same principle as secret zero — the *decryption* key
|
||||
is the out-of-band thing — applied to the handful of values that can't wait for the operator to spin up.
|
||||
|
||||
## The shape of it
|
||||
|
||||
The win isn't any one tool. It's the shape: **exactly one** credential lives outside GitOps, it's read-only
|
||||
and per-cluster, and it's the seed the whole tree grows from. Everything else is a pointer you can show
|
||||
your worst enemy. When someone asks "where are your secrets?", the honest answer is "in the store — the repo
|
||||
just knows their *names*," and that one sentence is the whole security model.
|
||||
|
||||
*This runs across a five-cluster Talos homelab; the ESO + Infisical wiring, the per-cluster read-only
|
||||
identities, and the revoke-order discipline are the load-bearing pieces.*
|
||||
Reference in New Issue
Block a user