---
title: "Powered off is a state, not an incident"
date: 2026-08-02
summary: "Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night, and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear."
tags: ["homelab", "operations", "talos", "sustainability", "architecture"]
draft: false
hero: "/blog/sleep-half-the-fleet.webp"
heroAlt: "A dark tower of circuit-traced architecture at night - most windows unlit and resting, a few glowing where the core still works"
---

Homelabs run 24/7 out of habit. The habit comes with a scoreboard - `uptime` counters climbing into the
hundreds of days, worn as proof of seriousness, and with two feelings nobody says out loud: the guilt of
turning a server off, as if a dark machine means you weren't serious; and the quieter fear that if you do
turn it off, it won't come back. Somewhere on that box, the fear whispers, is a state nobody wrote down,
held together by warmth.

Meanwhile the actual work is bursty. GPU inference runs when a request arrives. Transcode runs in batches
and finishes. The hardware idles around the clock for workloads that occupy it a fraction of the time - real watts, real fan-hours, real heat, spent keeping silicon warm for nobody.

Last night I shut down half the fleet on purpose. Three machines - two of them carrying an RTX 5080 each - were upgraded in the afternoon, verified healthy in the evening, and powered off before midnight. Nothing
broke, nothing was lost, and the fleet's books record the event as a state, not a failure. Think of a
building at night: the porter walks the floors, the lights go off in the wings nobody's using, and the
building isn't failing. It's resting, and the porter writes it in the log.

<!-- DIAGRAM: three node power states in a row - READY (lit) <-> OFF-BY-CHOICE (dark, a single power-button arrow back to Ready) and, set apart, PARKED-BROKEN (dark, a screwdriver arrow as its only exit); beneath all three states one continuous, unbroken document layer (git: machine config / CNI inline / workloads / credentials) - the document layer is what's highlighted, the power states sit on top of it as incidental -->
![One document persists beneath every power state - Ready, off by choice, parked broken](/diagrams/sleep-half-the-fleet.svg)

## Make the node a document, and power becomes a detail

None of this is safe by default. It's safe because of a property I argued for in [the sibling
post](/blog/no-ssh): each of these machines is a single-node Talos cluster whose entire existence is
declared. The machine config pins the disks by serial and the network interfaces by MAC; the CNI ships
inside that same document as an inline manifest; the workloads arrive from git the moment the node reports
Ready, reconciled by the fleet's GitOps controller; the credentials are substituted in at render time from
an encrypted file. There is no state that exists only on the box. The node *is* a document - power is just
whether it's currently being read.

That property is what made this week's sequence unremarkable. Three nodes had their network datapath
swapped - kube-proxy out, Cilium's eBPF replacement in, and their registry credentials rotated, one reboot
each. Every check came back green: GitOps fully synced, secrets flowing, every endpoint answering. And
then, the same night, all three were shut down. Upgraded, then slept. Nothing about the upgrade is lost to
the power state: it's on disk, in the config, in git. When the power button is next pressed, each node
boots into exactly the machine its documents describe - including everything that changed yesterday.

How they went down matters as much as the fact that they could. `talosctl shutdown` is a clean, API-driven
stop: workloads terminated properly, etcd closed cleanly, the platform powering itself off - confirmed dark
before anyone walked away. A graceful shutdown is what makes the next boot boring. Pull the plug instead
and you've converted a healthy state into a small forensic exercise for future-you.

## Keep the two kinds of dark apart

Four of the six machines are dark tonight, and the fleet's records refuse to treat them as one category.

Three (robin, kate and wgirl) are **off by choice**: healthy at shutdown, upgraded that same day, one
press of a power button from Ready. The fourth, elfastc, is **parked broken**: dark because of a hardware
fault, waiting on a screwdriver, not a switch. From the network's point of view the four are identical: no
link light, no metrics, nothing to distinguish rest from wreckage. From the operator's they are different
states with different exits, and the roster records which is which, in the same versioned pages as
everything else.

This is the porter's log. A dark wing and a broken lift look the same from the street; the difference is
written down at the desk. Six months from now, "why is this node off?" must have an answer that isn't
archaeology, because a fleet that can't tell *resting* from *broken* will eventually treat one as the
other, and both mistakes are expensive: rebuilding a healthy node because nobody trusted it, or trusting a
broken one because it merely looked asleep.

## Pay the costs where you can see them

Sleep isn't free, and pretending otherwise is how the 24/7 habit wins the argument. Two costs are real, and
both are stated in the books.

First: what lives on a sleeping node is dormant, not migrated. The fleet's LLM backend lives on wgirl's
GPU, and it sleeps when she does. The assistant app that depends on it stays up - it runs on a node that's
awake, but until power-on it has no model to call. That is a genuine loss of capability, accepted
deliberately, recorded next to the state that caused it. If the trade ever stops being worth it, the fix is
a power button, not a redesign.

Second: monitoring fires as a node goes silent. The metrics stream stops mid-sentence and the alert raises,
exactly as it should - the monitoring has no way to know the silence was chosen, and it must never assume
so. The wrong response is to call that a false alarm. It isn't. It's the system being honest about a
deliberate act, and the deliberate act is written down where the alert can be checked against it. The
porter doesn't disable the alarm panel for the dark wings - he checks the panel against the log.

## Design for off

Design for off. A node you can power down without ceremony is a node whose whole truth lives outside it - in documents, in git, in the log, and that is precisely the property you need on the day a node goes down
without asking. The fleet you can put to sleep on purpose and the fleet that shrugs off losing a node in
anger are the same fleet; the two abilities are one property. Uptime measures how long a machine has been
running. It says nothing about how well you hold it. A building isn't failing when its lights go out at
night - it's failing when nobody dares touch the switch.

*Live in the fleet's books tonight: three of six Talos nodes - robin and wgirl (an RTX 5080 each) and kate
(the AI sandbox) - upgraded to the new datapath, verified green, then gracefully shut down through the API
and recorded off-by-choice; elfastc parked-broken awaiting a screwdriver; cave and alfred carrying the
always-on core.*
