blog: publish five posts - landing-on-cilium, sleep-half-the-fleet, clone-not-keys, stage-the-locks, one-doorman-many-doors
Heroes optimised to 1600px webp from the prepped batch; diagrams authored to the house palette; dates set to publish day.
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: "The workbench gets a clone of the brain, never the keys"
|
||||
date: 2026-08-02
|
||||
summary: "A browser IDE you can reach from the couch or a phone is a legitimate want — and the obvious build, hosting it where the keys already live, quietly parks the whole estate behind a browser tab. Price the stolen session first, then hand the new surface copies of everything and originals of nothing, so the worst case is a git revert instead of a lost fleet."
|
||||
tags: ["security", "developer-experience", "sso", "kubernetes", "homelab"]
|
||||
draft: false
|
||||
---
|
||||
|
||||
|
||||
|
||||
You want your development environment everywhere. VS Code in a browser tab, reachable from the couch on the LAN and from a phone across the mesh VPN, with git and the daily tooling already inside. It's a legitimate want — it's how odd half-hours turn into finished work.
|
||||
|
||||
And the first instinct is obvious: host it on the operator box. That machine already has everything — every repo cloned, every context wired, every login warm. One container, one hostname, done by the weekend.
|
||||
|
||||
Read that instinct back slowly, because of what it actually proposes. The operator box is the one machine that can drive the entire fleet: the master decryption key for every secret, the admin kubeconfigs, the SSH keys. "Host it there" means parking all of that behind a browser tab — one stolen session away from anyone who ever finds the door. I've argued before that the fleet's keys should exist in exactly one place; this was the first time a convenience asked to move in with them. The design that survived the red-team gives the convenience something else entirely: a **clone** of the operator's brain — the repos, the tooling, the logins, furnished fresh. Never the keys.
|
||||
|
||||
<!-- DIAGRAM: browser + phone → authentik proxy (every path 302s to login, no carve-outs) → code-server pod [own volume: cloned repos, own git key, own logins — NO age key] → egress only DNS + gitea SSH + 443; the bastion (age key, admin kubeconfigs, SSH keys) sits outside the frame, unreachable. -->
|
||||
|
||||
|
||||
## Price a stolen session before you build the surface
|
||||
|
||||
The question that shaped this build wasn't "how do we secure it?" It was blunter: assume the worst has already happened — the session is fully stolen, an attacker sitting in the IDE as you — and write down what they now hold.
|
||||
|
||||
Against the first sketch, the one that ran on the operator host, the red-team returned three separate critical findings, all of them consequences of proximity to that host's keys. Each would have needed its own mitigation, its own monitoring, its own upkeep. Then the design moved to a disposable pod that inherits nothing from the operator host, and all three findings died in the same moment. Not mitigated — *deleted*. There was nothing left to steal along those routes, because nothing had been put there.
|
||||
|
||||
That is the cheapest security work I have ever done: three critical findings closed by a relocation, before a single line of configuration existed. Blast radius is a design input. Treat it as an audit output and you'll be retrofitting forever.
|
||||
|
||||
## Clone everything, inherit nothing
|
||||
|
||||
Think of the workbench as a reading room attached to an archive. A reading room is genuinely useful — every document laid out on the desks, annotated, workable, open late. But the pages on the desks are photocopies. The originals stay in the vault, and no reader's card opens it.
|
||||
|
||||
The pod starts empty. On first run I furnished it the way you'd stock a reading room: cloned the repos *into* it, generated a fresh git key *inside* it — registered in its own name, revocable on its own — and signed in to the daily tooling so those logins persist on the pod's own volume. Nothing is mounted from the operator host. Nothing is inherited. And the one thing deliberately kept out is the master secrets key — the key that turns the encrypted blobs in those repos into live fleet credentials. The workbench holds a copy of every document; the pages that matter are ciphertext to it.
|
||||
|
||||
One call in there deserves honesty, because it was a real trade. That git identity has account-level access: the workbench can push, including to the repo the fleet reconciles from. That is genuine power in a convenience surface, and I granted it with eyes open — partly because the door in front of it is layered, and mostly because of what the worst case *is*. A hostile push is loud. It lands in git history, attributed and timestamped, and `git revert` undoes it. Set that against the alternative worst case — silent possession of every credential in the estate — and "attacker can push to some repos" is a failure you recover from, audit, and learn from. The other one is a failure you rebuild from.
|
||||
|
||||
## Put everything behind one proven door
|
||||
|
||||
The front of the workbench is SSO-only, through the identity provider's proxy — and *everything* means everything. The route sends 100% of paths through the proxy, with no carve-outs, because the obvious carve-out is a hole: the terminal speaks over a WebSocket, and a WebSocket path excused from authentication is an unauthenticated shell. So it was proven, not assumed — probe the root unauthenticated, probe the terminal's WebSocket, probe the static assets and the health endpoint, and every one of them 302s to the login page.
|
||||
|
||||
Behind the SSO sits the application's own password, injected from the secrets manager rather than living in any repo. Belt behind braces: a bug in the proxy still lands on a locked application.
|
||||
|
||||
Under both sits a default-deny network policy. The pod can reach the git server, DNS, and its API endpoints over 443 — nothing else. That floor is real enough to bite: the git server's own policy had to grant the workbench an ingress seat by name, and the very first clone hung until it did. The hang *was* the policy working.
|
||||
|
||||
Just as important are the doors that don't exist. A direct LAN bind and a separate mesh-VPN serving path were both sketched, and both cut — the one guarded door already serves the couch and the phone alike. Every additional entrance is an additional thing to prove, forever.
|
||||
|
||||
## The principle
|
||||
|
||||
Every new convenience surface — a browser IDE, a remote dashboard, a sync agent — should be designed backwards from one number: what a *total* compromise of it yields. Compute that before the surface exists, while the answer is still a design choice. Then engineer it down: clones instead of originals, an identity scoped to the surface and revocable without ceremony, doors layered so no single failure is enough. Only when the worst case is something you could live through do you get to enjoy the convenience — and then you get to enjoy it completely, from any couch you like.
|
||||
|
||||
A reading room earns its comfort by holding photocopies. You can deface every page on every desk and the archive survives — because the originals were never in the room.
|
||||
|
||||
*Live at code.bztmon.org: a code-server pod on one Talos node, its home a 20Gi volume of cloned repos, its own git key and its own logins — behind an Authentik proxy that 302s every path, WebSockets included, to login, above a default-deny network policy. The fleet's age key has never entered it.*
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: "The best migrations delete more than they install"
|
||||
date: 2026-08-02
|
||||
summary: "Nobody chose our network stack — it accreted one default at a time, and every component was one more thing to version, patch, debug and alert on. Landing on Cilium wasn't an adoption; it was a demolition schedule, and the stack left standing is smaller than the one we started with."
|
||||
tags: ["cilium", "kubernetes", "networking", "architecture", "homelab"]
|
||||
draft: false
|
||||
---
|
||||
|
||||
|
||||
|
||||
Ask why the fleet ran flannel and there was never an answer worth writing down. Nobody chose it. Talos bundles flannel, so flannel ran. Kubernetes ships kube-proxy, so kube-proxy ran. LoadBalancer services need something on the LAN to answer for their addresses, so MetalLB moved in. HTTP needed terminating, so ingress-nginx went up, and later Traefik replaced it — on one cluster, Traefik still answers on ingress-nginx's old IP, the fossil of a hop nobody remembers deciding.
|
||||
|
||||
That's how most platform stacks are built. Not designed — accreted, the way an old house grows. A lean-to here, a carport there, each extension the quickest answer to the question of the day. And every extension arrives with its own roofline: a version to pin, a changelog to read, failure modes to learn, alerts to write.
|
||||
|
||||
This week the fleet finished landing on Cilium, and the honest record of that landing is the point of this post. It wasn't a product evaluation. It wasn't an adoption. It was a demolition schedule — and the stack left standing is smaller than the one we started with.
|
||||
|
||||
<!-- DIAGRAM: before/after columns on one cluster. Left (before): five separately versioned boxes — flannel (CNI), kube-proxy (service path), MetalLB (VIP announcement), Traefik/ingress-nginx (HTTP), and a dashed "NetworkPolicies — authored, unenforced" box — each with its own version tag. Right (after): a single Cilium box carrying CNI + eBPF service path + LB-IPAM/L2 announcements + Gateway API + enforced policy + Hubble flows, shipped inside the Talos machine document. The transition arrows are strike-throughs (deletions), not installs. -->
|
||||
|
||||
|
||||
## Count the rooflines, not the features
|
||||
|
||||
The starting shape, across a fleet of single-node Talos clusters: flannel carrying pod traffic, kube-proxy rewriting every service address, MetalLB answering ARP for the LoadBalancer VIPs, Traefik terminating HTTP. Plus a set of network policies that were authored, committed and completely inert — flannel can't enforce them, so the plans existed and nothing was ever built to them.
|
||||
|
||||
Four separate projects for four concerns, and a fifth concern going unmet. Each project a chart or a DaemonSet with a version, a CVE feed, an upgrade cadence, and its own distinct way of ruining an evening. None of them wrong individually. All of them defaults — and a default is a decision someone else made, for a fleet they've never seen.
|
||||
|
||||
What started the demolition wasn't "Cilium is better than flannel". It was noticing that one engine could carry all five concerns — the pod network, the service path, the VIP announcements, the policy enforcement, the flow visibility — and that one cluster in the fleet already ran that engine, a quiet standing proof. Four rooflines could become one.
|
||||
|
||||
## Swap the frame under an occupied house
|
||||
|
||||
Stage one, June: replace the CNI in place, on live clusters, no rebuilds. Per cluster the sequence is short. Set `cni: none` in the machine configuration, one reboot, install Cilium 1.19 with a pinned set of values, delete the flannel DaemonSet, then restart every non-hostNetwork pod so it leaves flannel's address management for Cilium's. Order the clusters lowest blast radius first; the control-plane hub goes last, once the procedure has become boring.
|
||||
|
||||
Two disciplines mattered more than the sequence. First, confirm the node rebooted at all — the apply returns immediately and the API often answers before the machine has cycled, so trust the uptime counter, not reachability. Second, those inert policies stop being inert the moment Cilium arrives: lines on a plan under flannel, enforced at the instant of cutover. Audit every allow-list *before* migrating the cluster that holds it.
|
||||
|
||||
Four live clusters converted in a day. Zero rebuilds, zero data loss, and the fleet verified green afterwards — 98 of 98 GitOps applications healthy. Then stage two, the reason enforcement mattered at all: a default-deny floor in every namespace, rolled fleet-wide with the observe-first method that has [its own post](/blog/observe-first-deny-second) — watch the real flows, then deny everything else ([the why lives here](/blog/trust-nothing)).
|
||||
|
||||
## Then pull the extensions down
|
||||
|
||||
Stage three, July — finished this week. With Cilium carrying pod traffic everywhere, the remaining structures came down one by one.
|
||||
|
||||
kube-proxy went first. Cilium's eBPF datapath took over service routing entirely, reaching the Kubernetes API through Talos's KubePrism endpoint on `localhost:7445`, after which the kube-proxy DaemonSet was deleted outright. Then MetalLB: Cilium's LB-IPAM handed out the same VIPs from the same pools, L2 announcements answered the same ARP queries, and every service kept its address through the swap. One reboot per node. On the media node, Cilium's Gateway API absorbed the ingress role as well — one more structure gone. MetalLB is now extinct across the fleet. Not deprecated, not legacy-mode: extinct.
|
||||
|
||||
The last demolition was managerial. The CNI no longer belongs to Helm at all — it ships as a Talos inline manifest, which means the network lives inside the machine's own configuration document. A rebuilt node comes up with its network the way it comes up with its kernel: because the document says so. Zero Helm release secrets remain anywhere in the fleet.
|
||||
|
||||
## Two structures fought back
|
||||
|
||||
There is always a price, and this one came due twice.
|
||||
|
||||
The first trap is the nastiest kind of failure: the change that succeeds and does nothing. Deleting Helm's release record orphans the resources, but it does *not* release Helm's per-field ownership inside them. So the flip to the eBPF datapath applied cleanly, the node rebooted on schedule — and `KubeProxyReplacement` still read `False`, because a config field Helm still owned silently refused the new manager's apply. No error. Nothing in the diff. It fired on every single cluster we flipped, so it's now a standard step rather than a contingency: steal the whole manifest's ownership server-side —
|
||||
|
||||
```
|
||||
kubectl apply --server-side --force-conflicts --field-manager=talos -f cilium-rendered.yaml
|
||||
```
|
||||
|
||||
— then verify the config changed before believing anything else.
|
||||
|
||||
The second was musical chairs with the VIPs. Mid-swap on one cluster, a service without a pinned address grabbed another service's freed IP the instant the old announcer released it — the LLM backend ended up sitting on the ingress controller's address. The fix is an ordering rule: pins first. Land the address pins while the old announcer is still alive, confirm they hold, and only then retire it. Two commits, in that order, every time.
|
||||
|
||||
## The principle
|
||||
|
||||
Consolidation is usually sold on the new tool's feature list. That's the wrong ledger. The value of landing on one engine wasn't what Cilium added — it was what the landing let us remove: a CNI, a service proxy, a VIP announcer, and on one node an ingress controller, each of which had been a version to pin, a feed to watch, an alert to tune, a way to be paged. Policy enforcement and flow observability never needed projects of their own; they came up with the same engine, side effects of the consolidation rather than line items on it.
|
||||
|
||||
Every structure you tear down is one that can never fall on you. Judge the next migration by its demolition list.
|
||||
|
||||
*Live across six single-node Talos clusters: Cilium 1.19.4 shipped as a Talos inline manifest, the eBPF datapath where kube-proxy used to be, LB-IPAM answering every VIP MetalLB once held, and Hubble watching the flows on every cluster.*
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
title: "SSO is for people, not for machines"
|
||||
date: 2026-08-02
|
||||
summary: "When an app grows a human face on an interface machines also depend on, one auth layer over everything either breaks the machines or quietly weakens the human door to let them through. The pattern that holds: a single identity-provider proxy in front of every human door, while machine paths keep their own scoped credentials — route by audience, not by app."
|
||||
tags: ["security", "sso", "authentik", "kubernetes", "homelab"]
|
||||
draft: false
|
||||
---
|
||||
|
||||
|
||||
|
||||
Sooner or later, a machine-facing service grows a human face. The container registry every node in the fleet pulls from ships a web UI. The development environment becomes a browser tab with a live terminal in it. Now there are people arriving at hostnames that machines also depend on — and the tempting move is one auth layer over the whole thing. Put the entire host behind single sign-on. One rule, no exceptions, done.
|
||||
|
||||
Do that to a registry and the cluster stops pulling images the same afternoon, because a container runtime cannot answer an interactive login page. The failure after that one is quieter and worse: you see the machines breaking, so you loosen the layer to let them through — an exception here, an anonymous path there — until the human door is weaker *because* the machines needed in.
|
||||
|
||||
The answer isn't a cleverer single layer. It's a split: **route by audience, not by app**. People get one doorman. Machines get keyed doors of their own, nowhere near the desk.
|
||||
|
||||
<!-- DIAGRAM: One hostname, two audiences. Left: human browsers converge on a single identity-provider proxy outpost (the doorman), which fronts multiple app doors — registry web UI and browser IDE — each app a provider bound to the SAME outpost. Right: machine clients (node container runtimes, CI) hit the registry's /v2 path directly, split off at the gateway, presenting their own scoped read-only credential — never touching the SSO layer. Show the gateway making the split: /v2 → registry direct, catch-all → outpost. Cyan #0c8fce human path, magenta #c026d3 machine path, violet #7c3aed app boxes, on the light card. -->
|
||||
|
||||
|
||||
## A doorman is for doors that can't answer for themselves
|
||||
|
||||
The house rule comes before anything else: when an app speaks OIDC natively, wire it natively. The app authenticates its own users against the identity provider and stays directly reachable — which matters, because its existing machine clients (a password manager's mobile app, a git CLI pushing over HTTPS) keep working untouched. Native wiring routes by audience for free.
|
||||
|
||||
A proxy outpost — an authenticating reverse proxy operated by the identity provider — is for the remainder: apps with no serious auth story of their own, or surfaces you'd never trust to defend themselves. Two tenants in this fleet fit that bill. A registry web UI, whose built-in auth is a static password file — a fine key for a machine, a poor front door for a person. And a browser IDE — an editor in a tab with a real shell behind it, which is about the most consequential thing you can put behind a URL.
|
||||
|
||||
## One desk signs in every visitor
|
||||
|
||||
Recent versions of the identity provider dropped their embedded proxy, so the outpost is now a deployment you run yourself. That sounds like a tax. It's the opposite, because you only ever run one. A single outpost serves many providers: each new application binds a new provider to the *same* instance. When the browser IDE arrived — the second tenant — the marginal cost was a provider object, a route pointing its hostname at the outpost, and a matching pair of network policies. No new deployment. App N+1 is a nameplate at the desk, not a second desk.
|
||||
|
||||
Standing up the desk the first time is where the hours went. Three traps, each with a price paid in real time:
|
||||
|
||||
- **A provider created by automation isn't a provider created by the UI.** Scripting the provider into existence skipped the OAuth defaults the admin UI sets silently — the redirect URIs sat empty, and every successful login dumped the user onto the identity provider's own homepage with a shrug of "no provider url". The login worked; it just went nowhere. Set the defaults explicitly, every time.
|
||||
- **The outpost has to be told which host the browser sees.** Left alone it knows only its internal service name, and it will happily redirect your browser to an address that exists nowhere outside the cluster. One environment variable carrying the browser-facing hostname closes the loop — and it wins over anything set through the API.
|
||||
- **A token with an invisible trailing newline fails like everything else fails.** The outpost's token, pasted into a secret, carried a newline you cannot see. The authorisation header it produces is invalid, and the symptoms are indistinguishable from both traps above. Strip the newline before the secret ever exists.
|
||||
|
||||
## Couriers don't queue at reception
|
||||
|
||||
The registry is the sharper case, because its real customers were never people. Every node's container runtime pulls images through its API — the `/v2` path — dozens of times a day. Put the doorman in front of that and every pull in the fleet dies waiting on a login page no runtime can answer.
|
||||
|
||||
So the route splits at the gateway. `/v2` goes straight to the registry. Everything else on the same hostname — the UI, the search — goes to the outpost and gets the full sign-in. And the direct path isn't left swinging open as the price of working: it's keyed with its own scoped, read-only identity, carried in each node's registry configuration. The couriers don't queue at reception, and reception doesn't prop the dock open for them either.
|
||||
|
||||
One hostname, two audiences, two doors — and neither is weakened to accommodate the other.
|
||||
|
||||
## Rattle every door before you trust the lock
|
||||
|
||||
The proof discipline matters as much as the pattern. The easy check is to load the homepage, watch it bounce to the sign-in page, and call the thing secured. But the homepage is the door nobody breaks in through.
|
||||
|
||||
The browser IDE made this concrete. Its whole value is a terminal speaking over a WebSocket — and a WebSocket doesn't render a login page. It either connects or it doesn't. Had the route carved that path out for convenience, the front door would be locked while a side corridor ran straight past the desk, exactly where no identity is ever checked. The same goes for static assets, and for the health endpoint.
|
||||
|
||||
So the route sends every path on the host to the outpost, and the check was empirical: an unauthenticated request to the root, to the terminal WebSocket, to a static asset and to the health endpoint each came back as a redirect to the identity provider. All of them. On a human door, a single carve-out *is* the hole; carve-outs belong only on machine doors, where they get keys of their own. And behind the doorman, the IDE still keeps its own password — the desk adds a layer; it doesn't replace the lock on the office.
|
||||
|
||||
## The principle
|
||||
|
||||
Slice authentication by who arrives, not by what serves the request. Humans are one audience — interactive, browser-borne, phishable — and they get a doorman: one strong flow, every human door in the estate behind the same desk, so door N+1 inherits the whole policy for the cost of a nameplate. Machines are the other audience — headless, scoped, incapable of answering a challenge — and they get narrow keys to their own doors, which the doorman never learns about.
|
||||
|
||||
The one-layer instinct fails because it treats the app as the unit of security. The unit is the audience. Sign the people in at one desk, key the machines at the dock — and neither door ever has to weaken to let the other's traffic through.
|
||||
|
||||
*Live in the homelab: one Authentik proxy outpost on the cave cluster fronting the Zot registry UI and a code-server IDE; the registry's `/v2` pull path keyed separately with a read-only identity across six Talos nodes.*
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "Powered off is a state, not an incident"
|
||||
date: 2026-08-02
|
||||
summary: "Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night — and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear."
|
||||
tags: ["homelab", "operations", "talos", "sustainability", "architecture"]
|
||||
draft: false
|
||||
---
|
||||
|
||||
|
||||
|
||||
Homelabs run 24/7 out of habit. The habit comes with a scoreboard — `uptime` counters climbing into the
|
||||
hundreds of days, worn as proof of seriousness — and with two feelings nobody says out loud: the guilt of
|
||||
turning a server off, as if a dark machine means you weren't serious; and the quieter fear that if you do
|
||||
turn it off, it won't come back. Somewhere on that box, the fear whispers, is a state nobody wrote down,
|
||||
held together by warmth.
|
||||
|
||||
Meanwhile the actual work is bursty. GPU inference runs when a request arrives. Transcode runs in batches
|
||||
and finishes. The hardware idles around the clock for workloads that occupy it a fraction of the time —
|
||||
real watts, real fan-hours, real heat, spent keeping silicon warm for nobody.
|
||||
|
||||
Last night I shut down half the fleet on purpose. Three machines — two of them carrying an RTX 5080 each —
|
||||
were upgraded in the afternoon, verified healthy in the evening, and powered off before midnight. Nothing
|
||||
broke, nothing was lost, and the fleet's books record the event as a state, not a failure. Think of a
|
||||
building at night: the porter walks the floors, the lights go off in the wings nobody's using, and the
|
||||
building isn't failing. It's resting — and the porter writes it in the log.
|
||||
|
||||
<!-- DIAGRAM: three node power states in a row — READY (lit) ⇄ OFF-BY-CHOICE (dark, a single power-button arrow back to Ready) and, set apart, PARKED-BROKEN (dark, a screwdriver arrow as its only exit); beneath all three states one continuous, unbroken document layer (git: machine config · CNI inline · workloads · credentials) — the document layer is what's highlighted, the power states sit on top of it as incidental -->
|
||||
|
||||
|
||||
## Make the node a document, and power becomes a detail
|
||||
|
||||
None of this is safe by default. It's safe because of a property I argued for in [the sibling
|
||||
post](/blog/no-ssh): each of these machines is a single-node Talos cluster whose entire existence is
|
||||
declared. The machine config pins the disks by serial and the network interfaces by MAC; the CNI ships
|
||||
inside that same document as an inline manifest; the workloads arrive from git the moment the node reports
|
||||
Ready, reconciled by the fleet's GitOps controller; the credentials are substituted in at render time from
|
||||
an encrypted file. There is no state that exists only on the box. The node *is* a document — power is just
|
||||
whether it's currently being read.
|
||||
|
||||
That property is what made this week's sequence unremarkable. Three nodes had their network datapath
|
||||
swapped — kube-proxy out, Cilium's eBPF replacement in — and their registry credentials rotated, one reboot
|
||||
each. Every check came back green: GitOps fully synced, secrets flowing, every endpoint answering. And
|
||||
then, the same night, all three were shut down. Upgraded, then slept. Nothing about the upgrade is lost to
|
||||
the power state: it's on disk, in the config, in git. When the power button is next pressed, each node
|
||||
boots into exactly the machine its documents describe — including everything that changed yesterday.
|
||||
|
||||
How they went down matters as much as the fact that they could. `talosctl shutdown` is a clean, API-driven
|
||||
stop: workloads terminated properly, etcd closed cleanly, the platform powering itself off — confirmed dark
|
||||
before anyone walked away. A graceful shutdown is what makes the next boot boring. Pull the plug instead
|
||||
and you've converted a healthy state into a small forensic exercise for future-you.
|
||||
|
||||
## Keep the two kinds of dark apart
|
||||
|
||||
Four of the six machines are dark tonight, and the fleet's records refuse to treat them as one category.
|
||||
|
||||
Three — robin, kate and wgirl — are **off by choice**: healthy at shutdown, upgraded that same day, one
|
||||
press of a power button from Ready. The fourth, elfastc, is **parked broken**: dark because of a hardware
|
||||
fault, waiting on a screwdriver, not a switch. From the network's point of view the four are identical — no
|
||||
link light, no metrics, nothing to distinguish rest from wreckage. From the operator's they are different
|
||||
states with different exits, and the roster records which is which, in the same versioned pages as
|
||||
everything else.
|
||||
|
||||
This is the porter's log. A dark wing and a broken lift look the same from the street; the difference is
|
||||
written down at the desk. Six months from now, "why is this node off?" must have an answer that isn't
|
||||
archaeology — because a fleet that can't tell *resting* from *broken* will eventually treat one as the
|
||||
other, and both mistakes are expensive: rebuilding a healthy node because nobody trusted it, or trusting a
|
||||
broken one because it merely looked asleep.
|
||||
|
||||
## Pay the costs where you can see them
|
||||
|
||||
Sleep isn't free, and pretending otherwise is how the 24/7 habit wins the argument. Two costs are real, and
|
||||
both are stated in the books.
|
||||
|
||||
First: what lives on a sleeping node is dormant, not migrated. The fleet's LLM backend lives on wgirl's
|
||||
GPU, and it sleeps when she does. The assistant app that depends on it stays up — it runs on a node that's
|
||||
awake — but until power-on it has no model to call. That is a genuine loss of capability, accepted
|
||||
deliberately, recorded next to the state that caused it. If the trade ever stops being worth it, the fix is
|
||||
a power button, not a redesign.
|
||||
|
||||
Second: monitoring fires as a node goes silent. The metrics stream stops mid-sentence and the alert raises,
|
||||
exactly as it should — the monitoring has no way to know the silence was chosen, and it must never assume
|
||||
so. The wrong response is to call that a false alarm. It isn't. It's the system being honest about a
|
||||
deliberate act, and the deliberate act is written down where the alert can be checked against it. The
|
||||
porter doesn't disable the alarm panel for the dark wings — he checks the panel against the log.
|
||||
|
||||
## The principle
|
||||
|
||||
Design for off. A node you can power down without ceremony is a node whose whole truth lives outside it —
|
||||
in documents, in git, in the log — and that is precisely the property you need on the day a node goes down
|
||||
without asking. The fleet you can put to sleep on purpose and the fleet that shrugs off losing a node in
|
||||
anger are the same fleet; the two abilities are one property. Uptime measures how long a machine has been
|
||||
running. It says nothing about how well you hold it. A building isn't failing when its lights go out at
|
||||
night — it's failing when nobody dares touch the switch.
|
||||
|
||||
*Live in the fleet's books tonight: three of six Talos nodes — robin and wgirl (an RTX 5080 each) and kate
|
||||
(the AI sandbox) — upgraded to the new datapath, verified green, then gracefully shut down through the API
|
||||
and recorded off-by-choice; elfastc parked-broken awaiting a screwdriver; cave and alfred carrying the
|
||||
always-on core.*
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
title: "Stage every lock before you turn one"
|
||||
date: 2026-08-02
|
||||
summary: "Turning off anonymous pulls on the registry a fleet boots from is a change that only fails at the next boot — and a node that can't pull at boot may not come back to tell you. So the flip is split in time: the credential staged inert in every node's config, each node rebooted and proven with a real pull, and the door closed only when the last key has turned."
|
||||
tags: ["security", "registry", "kubernetes", "talos", "homelab"]
|
||||
draft: false
|
||||
---
|
||||
|
||||
|
||||
|
||||
Every node in this fleet pulls its images through one registry: a pull-through mirror fronting five upstreams behind a single host. Anonymous read has been on since the day it went up — the bootstrap-era convenience you stop noticing, right up until you write the words *production posture* and realise the registry the entire estate boots from will hand an image to anyone who asks. The fix is not exotic: a named pull identity for the fleet, anonymous off.
|
||||
|
||||
And the reflex for shipping a fix like that is the reflex that works for everything else. Make the change, watch it, roll it back if it misbehaves. Flip the policy, delete a pod, watch the pod re-pull. Green means done.
|
||||
|
||||
Except this change doesn't fail when you make it. It fails at the *next boot* — and a node that can't pull at boot may not come back to tell you about it. No test run after the flip proves the fleet safe, because the only test that counts is a reboot, and the reboots that count are the ones you didn't plan. So the change gets split in time, the way a locksmith re-keys an occupied building: fit every cylinder and cut every key while the doors are propped open, try each key in its own door, and only when the last key turns do the props come out.
|
||||
|
||||
<!-- DIAGRAM: a three-stage timeline, left to right. STAGE: one pull credential fanned out inert into six node-config repos (git icon), the registry door propped open (anonymous read still ON). PROVE: per-node reboot onto the auth, then a gated pull of a mirror-only image through the node's own runtime — three of six ticked, one node parked for hardware repair. FLIP: anonymous read switched off at the registry ONLY at six of six — props out, door latches. Highlight the per-node gate, not the flip. -->
|
||||
|
||||
|
||||
## The fallback you'd bet on doesn't hold
|
||||
|
||||
On paper the naive flip looks survivable. Every node's registry config lists the mirror first and the true upstream second, so if the mirror answered a 401 you'd expect the runtime to shrug and pull direct. That expectation is exactly what the red-team refused to stake a fleet on: containerd's fallback behaviour on an auth failure is unreliable in practice — known, numbered upstream issues (#7321, #9997), not a hypothesis. And the trap has good manners. The flip goes green on the day, because every running pod already holds its images. The failure waits, patient, for the next power blip — then turns it into an outage.
|
||||
|
||||
So the red-team moved the flip to a window where every reboot is free: a fleet power-on, nodes coming up anyway, someone watching, a stumbling node getting attention instead of becoming a mystery at two in the morning. That's the first half of the split — this class of change never gets to surface its failure on a reboot you didn't choose.
|
||||
|
||||
## Cut every key while the doors are propped
|
||||
|
||||
The second half is staging, and staging happens well before enforcement. The credential exists now. It lives in an encrypted environment file beside each cluster's config and is substituted into the node's machine configuration when the config renders — the committed file carries a placeholder, never the value. That auth block went into all six node repos in one reviewed pass, while anonymous read stayed on. Inert. A credential the registry never challenges you for does nothing at all; it sits in git, costing nothing, waiting.
|
||||
|
||||
Two costs are worth naming while it waits. Committing is free but *loading* isn't: on this platform, mirror endpoints reload live while registry auth only takes effect at a reboot — which is precisely why the flip has to ride reboots that were already happening. And the rotation bill went into the decision record at decision time: this is one shared pull identity across six nodes, so rotating it later means six config edits and a fleet of reboots. Accepted, consciously, in writing — not discovered mid-incident two years from now.
|
||||
|
||||
## Try each key in its own door
|
||||
|
||||
Then each node proves itself, one at a time. The gate is not "it booted". Booting proves nothing — the doors are still propped, so anonymous pulls succeed regardless, and cached images would mask a broken credential anyway. The gate is a real pull, driven through the node's own container runtime via the management API, for an image that exists only on the mirror and nowhere upstream. A pass means one thing: this node presented the new credential and the registry honoured it. No fallback can fake that result, and no cache can serve it.
|
||||
|
||||
That's the discipline in a sentence: readiness is never inferred from the node coming back. Readiness is the key turning in the door, observed, per node.
|
||||
|
||||
## The props stay in until the last key turns
|
||||
|
||||
Where it stands, honestly: three of the six nodes are staged and gated as of this week, all in a single window — and cheaply, because each node's credential reboot rode a reboot that another migration was paying for anyway. Changes that share a reboot cost less than changes that each demand their own; batching them into one window is among the cheapest wins fleet operations offers.
|
||||
|
||||
The flip itself is still blocked. Deliberately. One node is parked awaiting a hardware repair, so the door stays propped for everyone until that node passes its gate too. That isn't the plan slipping — it *is* the plan. The asymmetry does all the work: a staged lock you haven't turned costs nothing, while a turned lock with one unproven key costs a boot loop on the worst morning of the quarter. With arithmetic that lopsided, waiting isn't caution. It's reading the numbers.
|
||||
|
||||
## The principle
|
||||
|
||||
Registry auth is one instance of a wider class: changes whose failure surfaces only at the next boot, restart, or failover — where the machine that would report the failure is the machine the failure takes down. For that class, make-it-and-watch is not a strategy, because there is nothing to watch until rollback is no longer on the table. The split is the strategy. Stage the change inert, in config, where it can be reviewed and reverted for free. Prove it per node, under conditions where failure is cheap and attended. Enforce only on full coverage — never on most of it.
|
||||
|
||||
Stage the locks first; turn them last.
|
||||
|
||||
*Live on a six-node Talos fleet: a zot pull-through mirror fronting five upstreams at one host, the fleet pull credential staged by sops-encrypted substitution into all six cluster repos, three nodes rebooted and gated with a mirror-only image pull — and anonymous read still on, deliberately, until the sixth key turns.*
|
||||
Reference in New Issue
Block a user