blog: publish five posts - landing-on-cilium, sleep-half-the-fleet, clone-not-keys, stage-the-locks, one-doorman-many-doors

Heroes optimised to 1600px webp from the prepped batch; diagrams authored
to the house palette; dates set to publish day.

src/content/blog/sleep-half-the-fleet.md

@@ -0,0 +1,98 @@
+---
+title: "Powered off is a state, not an incident"
+date: 2026-08-02
+summary: "Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night — and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear."
+tags: ["homelab", "operations", "talos", "sustainability", "architecture"]
+draft: false
+---
+
+![A dark tower of circuit-traced architecture at night — most windows unlit and resting, a few glowing where the core still works](/blog/sleep-half-the-fleet.webp)
+
+Homelabs run 24/7 out of habit. The habit comes with a scoreboard — `uptime` counters climbing into the
+hundreds of days, worn as proof of seriousness — and with two feelings nobody says out loud: the guilt of
+turning a server off, as if a dark machine means you weren't serious; and the quieter fear that if you do
+turn it off, it won't come back. Somewhere on that box, the fear whispers, is a state nobody wrote down,
+held together by warmth.
+
+Meanwhile the actual work is bursty. GPU inference runs when a request arrives. Transcode runs in batches
+and finishes. The hardware idles around the clock for workloads that occupy it a fraction of the time —
+real watts, real fan-hours, real heat, spent keeping silicon warm for nobody.
+
+Last night I shut down half the fleet on purpose. Three machines — two of them carrying an RTX 5080 each —
+were upgraded in the afternoon, verified healthy in the evening, and powered off before midnight. Nothing
+broke, nothing was lost, and the fleet's books record the event as a state, not a failure. Think of a
+building at night: the porter walks the floors, the lights go off in the wings nobody's using, and the
+building isn't failing. It's resting — and the porter writes it in the log.
+
+<!-- DIAGRAM: three node power states in a row — READY (lit) ⇄ OFF-BY-CHOICE (dark, a single power-button arrow back to Ready) and, set apart, PARKED-BROKEN (dark, a screwdriver arrow as its only exit); beneath all three states one continuous, unbroken document layer (git: machine config · CNI inline · workloads · credentials) — the document layer is what's highlighted, the power states sit on top of it as incidental -->
+![One document persists beneath every power state — Ready, off by choice, parked broken](/diagrams/sleep-half-the-fleet.svg)
+
+## Make the node a document, and power becomes a detail
+
+None of this is safe by default. It's safe because of a property I argued for in [the sibling
+post](/blog/no-ssh): each of these machines is a single-node Talos cluster whose entire existence is
+declared. The machine config pins the disks by serial and the network interfaces by MAC; the CNI ships
+inside that same document as an inline manifest; the workloads arrive from git the moment the node reports
+Ready, reconciled by the fleet's GitOps controller; the credentials are substituted in at render time from
+an encrypted file. There is no state that exists only on the box. The node *is* a document — power is just
+whether it's currently being read.
+
+That property is what made this week's sequence unremarkable. Three nodes had their network datapath
+swapped — kube-proxy out, Cilium's eBPF replacement in — and their registry credentials rotated, one reboot
+each. Every check came back green: GitOps fully synced, secrets flowing, every endpoint answering. And
+then, the same night, all three were shut down. Upgraded, then slept. Nothing about the upgrade is lost to
+the power state: it's on disk, in the config, in git. When the power button is next pressed, each node
+boots into exactly the machine its documents describe — including everything that changed yesterday.
+
+How they went down matters as much as the fact that they could. `talosctl shutdown` is a clean, API-driven
+stop: workloads terminated properly, etcd closed cleanly, the platform powering itself off — confirmed dark
+before anyone walked away. A graceful shutdown is what makes the next boot boring. Pull the plug instead
+and you've converted a healthy state into a small forensic exercise for future-you.
+
+## Keep the two kinds of dark apart
+
+Four of the six machines are dark tonight, and the fleet's records refuse to treat them as one category.
+
+Three — robin, kate and wgirl — are **off by choice**: healthy at shutdown, upgraded that same day, one
+press of a power button from Ready. The fourth, elfastc, is **parked broken**: dark because of a hardware
+fault, waiting on a screwdriver, not a switch. From the network's point of view the four are identical — no
+link light, no metrics, nothing to distinguish rest from wreckage. From the operator's they are different
+states with different exits, and the roster records which is which, in the same versioned pages as
+everything else.
+
+This is the porter's log. A dark wing and a broken lift look the same from the street; the difference is
+written down at the desk. Six months from now, "why is this node off?" must have an answer that isn't
+archaeology — because a fleet that can't tell *resting* from *broken* will eventually treat one as the
+other, and both mistakes are expensive: rebuilding a healthy node because nobody trusted it, or trusting a
+broken one because it merely looked asleep.
+
+## Pay the costs where you can see them
+
+Sleep isn't free, and pretending otherwise is how the 24/7 habit wins the argument. Two costs are real, and
+both are stated in the books.
+
+First: what lives on a sleeping node is dormant, not migrated. The fleet's LLM backend lives on wgirl's
+GPU, and it sleeps when she does. The assistant app that depends on it stays up — it runs on a node that's
+awake — but until power-on it has no model to call. That is a genuine loss of capability, accepted
+deliberately, recorded next to the state that caused it. If the trade ever stops being worth it, the fix is
+a power button, not a redesign.
+
+Second: monitoring fires as a node goes silent. The metrics stream stops mid-sentence and the alert raises,
+exactly as it should — the monitoring has no way to know the silence was chosen, and it must never assume
+so. The wrong response is to call that a false alarm. It isn't. It's the system being honest about a
+deliberate act, and the deliberate act is written down where the alert can be checked against it. The
+porter doesn't disable the alarm panel for the dark wings — he checks the panel against the log.
+
+## The principle
+
+Design for off. A node you can power down without ceremony is a node whose whole truth lives outside it —
+in documents, in git, in the log — and that is precisely the property you need on the day a node goes down
+without asking. The fleet you can put to sleep on purpose and the fleet that shrugs off losing a node in
+anger are the same fleet; the two abilities are one property. Uptime measures how long a machine has been
+running. It says nothing about how well you hold it. A building isn't failing when its lights go out at
+night — it's failing when nobody dares touch the switch.
+
+*Live in the fleet's books tonight: three of six Talos nodes — robin and wgirl (an RTX 5080 each) and kate
+(the AI sandbox) — upgraded to the new datapath, verified green, then gracefully shut down through the API
+and recorded off-by-choice; elfastc parked-broken awaiting a screwdriver; cave and alfred carrying the
+always-on core.*