copy pass: straighten quotes and arrows, thin the dash asides, drop the app-count caption from the constellation
build-and-deploy / build (push) Failing after 13m18s

This commit is contained in:
2026-08-17 23:12:42 +10:00
parent 78e8740408
commit 3cfd59a464
23 changed files with 257 additions and 271 deletions
-1
View File
@@ -109,7 +109,6 @@
<circle class="core core--cyan pulse" cx="380" cy="310" r="5"></circle>
<rect class="plate plate--hub" x="338" y="352" width="84" height="24" rx="4"></rect>
<text class="lbl lbl--hub" x="380" y="368" text-anchor="middle">cave</text>
<text class="cap cap--hub" x="380" y="392" text-anchor="middle">139 apps · reconciling</text>
<!-- ── awake: alfred (Intel compute — NUC iGPU/QuickSync) ── -->
<circle cx="612" cy="128" r="24" fill="rgba(0,199,253,0.07)"></circle>
+9 -10
View File
@@ -1,7 +1,7 @@
---
title: "SSO is a perimeter decision, not a login box"
date: 2026-06-23
summary: "Putting single sign-on in front of a fleet of self-hosted apps isn't about a nicer login screen it's about where your trust boundary lives, and how it fails."
summary: "Putting single sign-on in front of a fleet of self-hosted apps isn't about a nicer login screen - it's about where your trust boundary lives, and how it fails."
tags: ["authentik", "oidc", "sso", "security", "kubernetes"]
draft: false
hero: "/blog/authentik-sso-2fa.webp"
@@ -10,18 +10,17 @@ heroAlt: "A single central gate ringed by a second concentric seal radiates beam
---
Most write-ups treat single sign-on as a feature you bolt on: stand up an identity provider, point the
apps at it, enjoy one login. That framing misses the actual decision. SSO *moves your trust boundary*
every app now trusts one issuer, and every account is one credential away from all of them. Get it right
apps at it, enjoy one login. That framing misses the actual decision. SSO *moves your trust boundary* - every app now trusts one issuer, and every account is one credential away from all of them. Get it right
and you've hardened the whole estate at once. Get it wrong and you've built a single, central thing to lose.
Here's how I think about it after wiring an OIDC provider in front of a self-hosted fleet.
![Authentik SSO flow the browser authenticates against the identity provider over the front channel; each app validates tokens over an internal back channel](/diagrams/authentik-sso.svg?v=2)
![Authentik SSO flow - the browser authenticates against the identity provider over the front channel; each app validates tokens over an internal back channel](/diagrams/authentik-sso.svg?v=2)
## Enforce the second factor at the flow, not per app
The leverage is putting 2FA on the *authentication flow itself*, not on each application. Every app that
federates to the provider inherits it for free you can't forget to enable 2FA on app number nine, because
federates to the provider inherits it for free - you can't forget to enable 2FA on app number nine, because
app number nine never sees a password. Enrolment (TOTP / WebAuthn) is mandatory at the identity layer, and
the apps just receive an already-verified identity.
@@ -30,8 +29,8 @@ gets it.
## Keep a key under the mat
The flip side of "one issuer for everything" is that when the issuer is down or you fat-finger the auth
flow you can lock yourself out of *everything*, including the tools you'd use to fix it. So every critical
The flip side of "one issuer for everything" is that when the issuer is down, or you fat-finger the auth
flow - you can lock yourself out of *everything*, including the tools you'd use to fix it. So every critical
app keeps a **local break-glass admin** that bypasses SSO, and the recovery codes for 2FA enrolment live
offline.
@@ -43,19 +42,19 @@ Two traps ate real hours, and they're the same lesson wearing two hats:
- **A baked-in config file silently overrode environment variables.** I set the OIDC scopes via env, login
kept failing with *"missing fields: email"*, and the cause was a config file winning over the env. The
provider needs `openid email profile` explicitly set where the app actually reads it, not where you
provider needs `openid email profile` explicitly - set where the app actually reads it, not where you
assume it does.
- **In-cluster back-channels can't use the public hostname.** The browser hits the public login URL fine,
but the app's *server-side* token exchange runs inside the cluster, where split-horizon DNS doesn't
resolve the public name. The fix: point the back channel at the internal service address while the
browser-facing URL stays public.
Both say the same thing: SSO has two channels the one the user sees and the one the server uses and they
Both say the same thing: SSO has two channels - the one the user sees and the one the server uses, and they
don't live on the same network.
## The principle
Centralising identity is a force multiplier, but it concentrates risk *by design*. So design the
concentration: enforce the strong factor at the boundary, plan the failure mode before you need it, and
remember that OIDC authenticates a *user* it doesn't, on its own, make a sensitive service safe to expose.
remember that OIDC authenticates a *user* - it doesn't, on its own, make a sensitive service safe to expose.
The login box is the easy part. The perimeter is the decision.
+13 -14
View File
@@ -1,10 +1,10 @@
---
title: "The bastion is a pattern, not a box"
date: 2026-07-03
summary: "Admin credentials accumulate on whatever machine you happen to work from, until your daily driver is the real control plane of the fleet. A bastion fixes that not by being special hardware, but by being the one place the keys live and the discipline that they never leave it."
summary: "Admin credentials accumulate on whatever machine you happen to work from, until your daily driver is the real control plane of the fleet. A bastion fixes that, not by being special hardware, but by being the one place the keys live and the discipline that they never leave it."
tags: ["security", "operations", "homelab", "tailscale", "access"]
hero: "/blog/bastion-is-a-pattern.webp?v=1"
heroAlt: "A lone illuminated harbour pilot station on a dark sea, thin beams of cyan and magenta light reaching out to distant vessels the charts and keys stay at the station, the ships only transit past it."
heroAlt: "A lone illuminated harbour pilot station on a dark sea, thin beams of cyan and magenta light reaching out to distant vessels - the charts and keys stay at the station, the ships only transit past it."
draft: false
---
@@ -13,7 +13,7 @@ Here's the uncomfortable default: your fleet's admin credentials live wherever y
kubeconfig on the laptop, another copy on the desktop, an age key pasted onto a third machine "just for one
decrypt", SSH keys sprouting wherever a clone was convenient. Nobody decided this. It accumulates.
Which means your **daily driver is the real control plane** the machine that browses the web, opens email
Which means your **daily driver is the real control plane** - the machine that browses the web, opens email
attachments and runs whatever you installed last week is also the machine that can wipe a cluster. Lose it,
rebuild it, or compromise it, and you've done all three to the fleet.
@@ -26,14 +26,14 @@ never leave it. Not a hardened appliance. A pattern.
## Everything transits, nothing homes anywhere else
Think of a harbour pilot station. Every ship entering the harbour is guided by a pilot, but the charts and the
local knowledge never board the ships they stay at the station, and the vessels transit past it. That's the
local knowledge never board the ships - they stay at the station, and the vessels transit past it. That's the
whole design: the dangerous knowledge has exactly one home, and traffic comes *to* it.
Mine is nothing special on purpose: a small Debian VM four cores, no GPU passthrough, because a bastion needs
Mine is nothing special on purpose: a small Debian VM - four cores, no GPU passthrough, because a bastion needs
none. What makes it the bastion is what it holds. One kubeconfig with a context for every cluster. The
talosconfig for the OS layer beneath them. The SOPS age key that decrypts the fleet's secrets, sitting in
exactly one place at `~/.config/sops/age/keys.txt`. A checkout of every ops repo. And a toolchain pinned
deliberately `talosctl` matched to the fleet's live Talos version, because a drifted client against a
deliberately - `talosctl` matched to the fleet's live Talos version, because a drifted client against a
production node is how apply-time surprises happen.
None of that is exotic. The pattern is the *singularity* of it: ask "where can the fleet be driven from?" and
@@ -42,38 +42,37 @@ the answer is one hostname, and auditing that answer takes a minute.
## The bastion is defined by what it refuses
The keys that live there matter less than the keys that were *refused*. When this box was stood up to take
over the operator role from an older machine before that machine's wipe and rebirth as a GPU node the
over the operator role from an older machine - before that machine's wipe and rebirth as a GPU node - the
login credentials for my agent tooling were deliberately not copied across. Fresh auth on the new box beats a
credential that has now existed in two places, because the whole point collapses the moment "one place"
becomes "two, temporarily".
The stand-up also proved the pattern's diagnostic value. Every repo cloned fine and then a push failed,
The stand-up also proved the pattern's diagnostic value. Every repo cloned fine, and then a push failed,
because the git remotes resolved through SSH host aliases that only existed on the *old* machine. Access had
been quietly welded to one particular computer for months, and nothing noticed while that computer was always
there. Consolidating forces the question: is this credential something the operator point holds, or something
one box happens to have? The fix was the pattern applied properly a new key generated *on* the bastion,
one box happens to have? The fix was the pattern applied properly - a new key generated *on* the bastion,
registered once, never exported.
## The one that costs an evening
The bastion has to be reachable off-LAN, so it joined the tailnet as a subnet router, advertising its own
network. Here's the trap: it also sat *inside* a subnet another router already advertised and bringing it
network. Here's the trap: it also sat *inside* a subnet another router already advertised, and bringing it
up with `--accept-routes` made it accept the overlay route *for its own LAN*. Local replies went into the
tunnel instead of out the NIC. ARP stayed healthy, every TCP connection died, and SSH to the box that holds
all the keys went dark.
The rule that falls out: a node on the only LAN **advertises routes but never accepts them** it reaches that
The rule that falls out: a node on the only LAN **advertises routes but never accepts them** - it reaches that
network natively. And the recovery is itself a bastion lesson: you get back in *out-of-band*, through the
hypervisor console. The control point must sit on infrastructure you can still reach when the network lies.
## The principle
A bastion isn't a product you install; it's a property you enforce: **the set of places your keys exist has
exactly one member.** Any box can hold the role a VM, a spare NUC, a cloud instance and the role can move,
exactly one member.** Any box can hold the role - a VM, a spare NUC, a cloud instance, and the role can move,
the way this one inherited it from a machine headed for a wipe. What never moves casually is the material
itself. Ships come and go all day.
The charts never leave the station.
*Live as a four-core Debian VM at 10.0.11.11 one kubeconfig, one talosconfig, one age key, every ops repo
driving a five-cluster Talos fleet, reachable over the tailnet as its second subnet router.*
*Live as a four-core Debian VM at 10.0.11.11 - one kubeconfig, one talosconfig, one age key, every ops repo - driving a five-cluster Talos fleet, reachable over the tailnet as its second subnet router.*
+13 -13
View File
@@ -1,7 +1,7 @@
---
title: "An agent should never hold the key it's using"
date: 2026-07-03
summary: "You want an AI agent that can actually do things call APIs, touch real data. You also don't fully trust it. The resolution isn't a better sandbox; it's making sure the agent never possesses a credential at all. A broker holds the keys, mints short-lived capabilities, and gates every write behind a human. Here's the pattern."
summary: "You want an AI agent that can actually do things - call APIs, touch real data. You also don't fully trust it. The resolution isn't a better sandbox; it's making sure the agent never possesses a credential at all. A broker holds the keys, mints short-lived capabilities, and gates every write behind a human. Here's the pattern."
tags: ["security", "ai-agents", "architecture", "zero-trust", "homelab"]
draft: false
hero: "/blog/broker-pattern.webp"
@@ -11,8 +11,8 @@ heroAlt: "An untrusted agent reaches through a sealed gate to ask; on the far si
The previous post put untrusted code in a hardware-isolated VM, and ended on a caveat: isolation contains an
*escape*, but it does nothing about an agent **misusing a tool it was legitimately given**. If you hand an AI
agent a database credential so it can be useful, a single bad decision a prompt injection, a confused chain
of reasoning spends that credential. The sandbox did its job perfectly and you still got robbed, through the
agent a database credential so it can be useful, a single bad decision - a prompt injection, a confused chain
of reasoning - spends that credential. The sandbox did its job perfectly and you still got robbed, through the
front door you built.
So the real question isn't "how do I isolate the agent?" It's "how does the agent get work done *without ever
@@ -22,13 +22,13 @@ holding a key*?"
The pattern is to put every credential, every tool, and every model endpoint **behind a broker**, and give the
agent only a *capability to ask*. The agent never sees a token. It calls the broker; the broker holds the real
credential, decides whether the request is allowed, and if it is performs the action itself and returns the
credential, decides whether the request is allowed, and - if it is - performs the action itself and returns the
result. The key never leaves the broker.
That one inversion changes the threat model completely. A fully compromised agent can now do exactly one thing:
**ask**. And asking is answered by something it can't reach, can't impersonate, and can't bypass.
<!-- DIAGRAM: untrusted sandbox cred-broker (mint capability) mcp-broker (verify + run tool with held cred) human-approval gate downstream. Agent never touches the credential. -->
<!-- DIAGRAM: untrusted sandbox -> cred-broker (mint capability) -> mcp-broker (verify + run tool with held cred) -> human-approval gate -> downstream. Agent never touches the credential. -->
![The two-tier broker flow: capability in, action out, credential never exposed](/diagrams/broker-pattern.svg)
## Two clusters, two brokers
@@ -36,39 +36,39 @@ That one inversion changes the threat model completely. A fully compromised agen
The trust boundary is physical, not just logical. The **untrusted executor** (the Kata sandboxes) lives on one
cluster; the **trusted tier** (the brokers, the model gateway, the real credentials) lives on a *separate*
cluster. A total compromise of the executor still can't reach the brokers' secrets except across a policed
network link there's no shared kernel, no shared API server, nothing to pivot through.
network link - there's no shared kernel, no shared API server, nothing to pivot through.
On the trusted side there are two brokers, deliberately split:
- A **credential broker** validates the sandbox's identity (a short-lived, signed token unique to the task) and
mints a **capability** a cryptographically signed, scoped, single-use, expiring grant. Not a credential. A
mints a **capability** - a cryptographically signed, scoped, single-use, expiring grant. Not a credential. A
*permission to ask for one specific thing*.
- A **tool broker** takes that capability, verifies the signature, the scope, and the one-time nonce, and only
then runs the requested tool using a credential *it* holds. The result comes back; the credential doesn't.
then runs the requested tool - using a credential *it* holds. The result comes back; the credential doesn't.
## A human gates every write
Reads are one thing. For anything that *changes the world* creating, deleting, sending the broker doesn't
Reads are one thing. For anything that *changes the world* - creating, deleting, sending - the broker doesn't
just decide on policy. It **stages** the action and pings a human: an Approve/Deny prompt on my phone, carrying
a one-time token bound to the exact task, method, and arguments. Tap approve and it executes; tap deny, or
ignore it, and it doesn't. The gate is **fail-closed**: a timeout is a denial, a replayed token is rejected, an
unknown method is rejected. The default, always, is *no*.
The load-bearing principle underneath all of it: **enforcement that has to survive a compromised agent lives at
the broker, never at the agent or the orchestrator.** Approval, scope, budgets, write-authority none of it
the broker, never at the agent or the orchestrator.** Approval, scope, budgets, write-authority - none of it
lives anywhere the agent's reasoning can touch. The agent can be wrong, jailbroken, or outright hostile, and the
worst-case is still just *a request that gets refused*.
## Useful and safe at the same time
It's tempting to think you have to choose: give the agent real power and accept real risk, or lock it down so
hard it can't do anything. The broker pattern is how you get both. The agent is *useful* it can call real
tools against real systems. It's *safe* it never holds a key, every write waits on a human, and the moment
hard it can't do anything. The broker pattern is how you get both. The agent is *useful* - it can call real
tools against real systems. It's *safe* - it never holds a key, every write waits on a human, and the moment
something goes wrong, the blast radius is a denied request, not a spent credential.
Run agents like you'd run any other untrusted input: assume it's compromised, and make sure that assumption is
*boring*.
*Live as the trusted tier of a two-cluster AI-agent platform: a credential broker and a tool broker holding the
keys, a phone-based human-approval gate on every write, and an agent that by construction never sees a
keys, a phone-based human-approval gate on every write, and an agent that - by construction - never sees a
secret.*
+11 -11
View File
@@ -1,7 +1,7 @@
---
title: "The cert that couldn't see its own proof"
date: 2026-07-03
summary: "Let's Encrypt DNS-01 posted its proof to the public internet and then checked for it through the LAN's split-horizon DNS, which will never carry it. The fix is one line: make the self-check look where the CA looks, not where you live."
summary: "Let's Encrypt DNS-01 posted its proof to the public internet, and then checked for it through the LAN's split-horizon DNS, which will never carry it. The fix is one line: make the self-check look where the CA looks, not where you live."
tags: ["tls", "dns", "cert-manager", "lets-encrypt", "split-horizon"]
hero: "/blog/cert-couldnt-see-its-proof.webp?v=1"
heroAlt: "A sealed structure split by a translucent wall: a glowing proof-token sits in the open on the outside where a distant beam finds it, while an identical vantage on the inside looks toward the same spot and sees nothing."
@@ -9,13 +9,13 @@ draft: false
---
Here's the uncomfortable default: cert-manager checks its own homework **through whatever DNS the cluster hands it**. On a normal network that's fine the cluster's resolvers see the same internet everyone else does. On a split-horizon network, they don't. Your internal DNS is authoritative for the domain, answers first, and knows nothing about the record you just published to the world.
Here's the uncomfortable default: cert-manager checks its own homework **through whatever DNS the cluster hands it**. On a normal network that's fine - the cluster's resolvers see the same internet everyone else does. On a split-horizon network, they don't. Your internal DNS is authoritative for the domain, answers first, and knows nothing about the record you just published to the world.
So the DNS-01 challenge does something quietly absurd. cert-manager writes the `_acme-challenge` TXT record to Cloudflare and it lands. `dig @1.1.1.1` returns the token. Let's Encrypt could validate it right now. But before asking the CA to look, cert-manager runs a *self-check*: it resolves the record itself, through cluster DNS, through the Pi-hole that owns `bztmon.org` internally and has no such TXT record. Nothing comes back. So it waits, and checks again, and loops on `"not yet propagated"` forever.
So the DNS-01 challenge does something quietly absurd. cert-manager writes the `_acme-challenge` TXT record to Cloudflare, and it lands. `dig @1.1.1.1` returns the token. Let's Encrypt could validate it right now. But before asking the CA to look, cert-manager runs a *self-check*: it resolves the record itself, through cluster DNS, through the Pi-hole that owns `bztmon.org` internally and has no such TXT record. Nothing comes back. So it waits, and checks again, and loops on `"not yet propagated"` - forever.
It's posting a letter into the public postbox, then walking back inside to check the hallway mail slot for it. The letter is out there. The postman can see it. You're looking in the one box it will never reach and concluding it hasn't been sent.
It's posting a letter into the public postbox, then walking back inside to check the hallway mail slot for it. The letter is out there. The postman can see it. You're looking in the one box it will never reach, and concluding it hasn't been sent.
<!-- DIAGRAM: split-horizon DNS-01 cert-manager writes the TXT to Cloudflare (public path, record live), Let's Encrypt validates via public DNS, but the self-check arrow loops through the internal Pi-hole which returns nothing; the fix arrow reroutes the self-check to 1.1.1.1/8.8.8.8 -->
<!-- DIAGRAM: split-horizon DNS-01 - cert-manager writes the TXT to Cloudflare (public path, record live), Let's Encrypt validates via public DNS, but the self-check arrow loops through the internal Pi-hole which returns nothing; the fix arrow reroutes the self-check to 1.1.1.1/8.8.8.8 -->
![The self-check resolving through the wrong horizon while the CA sees the record](/diagrams/cert-couldnt-see-its-proof.svg?v=1)
## Look where the examiner looks
@@ -27,19 +27,19 @@ dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
dns01RecursiveNameserversOnly: true
```
That points the self-check at public resolvers the same vantage Let's Encrypt validates from instead of the cluster's own view of the world. The moment cert-manager checks the postbox the postman actually collects from, the challenge that had been pending for an age clears in seconds. No record changed. No DNS changed. Only the *observer* moved.
That points the self-check at public resolvers - the same vantage Let's Encrypt validates from - instead of the cluster's own view of the world. The moment cert-manager checks the postbox the postman actually collects from, the challenge that had been pending for an age clears in seconds. No record changed. No DNS changed. Only the *observer* moved.
One footnote if you run default-deny egress: the check now leaves the cluster on port 53, so `world:53` has to be allowed for the cert-manager pods. Here it already was, courtesy of the same policy that lets cloudflared out.
## Two proofs, one postbox
The same challenge has a second way to hang, and it looks identical from the outside. Request a certificate for both the wildcard `*.bztmon.org` *and* the apex `bztmon.org`, and Let's Encrypt opens two authorisations both of which validate at the **same** record name, `_acme-challenge.bztmon.org`, with *different* tokens. cert-manager's Cloudflare solver dutifully writes one, then the other solver overwrites it, then the first writes it back. Two letters fighting over one envelope, each self-check finding the other's contents. Both loop on `"not yet propagated"` and neither is a propagation problem.
The same challenge has a second way to hang, and it looks identical from the outside. Request a certificate for both the wildcard `*.bztmon.org` *and* the apex `bztmon.org`, and Let's Encrypt opens two authorisations - both of which validate at the **same** record name, `_acme-challenge.bztmon.org`, with *different* tokens. cert-manager's Cloudflare solver dutifully writes one, then the other solver overwrites it, then the first writes it back. Two letters fighting over one envelope, each self-check finding the other's contents. Both loop on `"not yet propagated"`, and neither is a propagation problem.
The fix is to stop asking for both: request the wildcard only, and drop the apex SAN unless you serve the bare domain. This one ate a real evening on robin before the pattern was obvious.
## A permission failure in a propagation costume
The third trap doesn't even mention permissions. A Cloudflare API token with `Zone:DNS:Edit` alone *looks* sufficient it can edit records, what more could a DNS solver want? But cert-manager first has to enumerate zones to find the zone ID, and that needs **`Zone:Zone:Read`**. Without it, the failure surfaces not as `403` but as you guessed it `"not yet propagated"`.
The third trap doesn't even mention permissions. A Cloudflare API token with `Zone:DNS:Edit` alone *looks* sufficient: it can edit records, what more could a DNS solver want? But cert-manager first has to enumerate zones to find the zone ID, and that needs **`Zone:Zone:Read`**. Without it, the failure surfaces not as `403` but as - you guessed it - `"not yet propagated"`.
Add Zone:Read to the token at Cloudflare and it unblocks instantly. The token string doesn't change, so no secret rotates; the fix never touches the cluster at all.
@@ -47,8 +47,8 @@ Three different root causes. One identical symptom. That's the real menace of th
## The principle
A self-check is only worth what its vantage point is worth. If your verifier stands outside a CA on the public internet, a monitor beyond the LAN, an auditor reading the published copy then a check that observes from *inside* isn't a weaker version of the truth. It's a different truth, and it can disagree indefinitely.
A self-check is only worth what its vantage point is worth. If your verifier stands outside - a CA on the public internet, a monitor beyond the LAN, an auditor reading the published copy - then a check that observes from *inside* isn't a weaker version of the truth. It's a different truth, and it can disagree indefinitely.
So when a system checks its own work before presenting it, ask one question first: does it look from where the judge will look? Check the postbox the postman collects from not the mail slot in your own hallway.
So when a system checks its own work before presenting it, ask one question first: does it look from where the judge will look? Check the postbox the postman collects from, not the mail slot in your own hallway.
*Bit the media-gateway wildcard on elfastc, 2026-07-02 cert-manager v1.19.5 issuing `*.bztmon.org` via Cloudflare DNS-01, on a fleet where the internal Pi-hole owns the zone and public resolvers hold the proof.*
*Bit the media-gateway wildcard on elfastc, 2026-07-02 - cert-manager v1.19.5 issuing `*.bztmon.org` via Cloudflare DNS-01, on a fleet where the internal Pi-hole owns the zone and public resolvers hold the proof.*
+16 -16
View File
@@ -1,52 +1,52 @@
---
title: "The workbench gets a clone of the brain, never the keys"
date: 2026-08-02
summary: "A browser IDE you can reach from the couch or a phone is a legitimate want and the obvious build, hosting it where the keys already live, quietly parks the whole estate behind a browser tab. Price the stolen session first, then hand the new surface copies of everything and originals of nothing, so the worst case is a git revert instead of a lost fleet."
summary: "A browser IDE you can reach from the couch or a phone is a legitimate want, and the obvious build, hosting it where the keys already live, quietly parks the whole estate behind a browser tab. Price the stolen session first, then hand the new surface copies of everything and originals of nothing, so the worst case is a git revert instead of a lost fleet."
tags: ["security", "developer-experience", "sso", "kubernetes", "homelab"]
draft: false
hero: "/blog/clone-not-keys.webp"
heroAlt: "A luminous reading room of copied pages, thin circuit traces running back to a sealed vault where the original documents rest untouched"
---
You want your development environment everywhere. VS Code in a browser tab, reachable from the couch on the LAN and from a phone across the mesh VPN, with git and the daily tooling already inside. It's a legitimate want it's how odd half-hours turn into finished work.
You want your development environment everywhere. VS Code in a browser tab, reachable from the couch on the LAN and from a phone across the mesh VPN, with git and the daily tooling already inside. It's a legitimate want - it's how odd half-hours turn into finished work.
And the first instinct is obvious: host it on the operator box. That machine already has everything every repo cloned, every context wired, every login warm. One container, one hostname, done by the weekend.
And the first instinct is obvious: host it on the operator box. That machine already has everything - every repo cloned, every context wired, every login warm. One container, one hostname, done by the weekend.
Read that instinct back slowly, because of what it actually proposes. The operator box is the one machine that can drive the entire fleet: the master decryption key for every secret, the admin kubeconfigs, the SSH keys. "Host it there" means parking all of that behind a browser tab one stolen session away from anyone who ever finds the door. I've argued before that the fleet's keys should exist in exactly one place; this was the first time a convenience asked to move in with them. The design that survived the red-team gives the convenience something else entirely: a **clone** of the operator's brain the repos, the tooling, the logins, furnished fresh. Never the keys.
Read that instinct back slowly, because of what it actually proposes. The operator box is the one machine that can drive the entire fleet: the master decryption key for every secret, the admin kubeconfigs, the SSH keys. "Host it there" means parking all of that behind a browser tab - one stolen session away from anyone who ever finds the door. I've argued before that the fleet's keys should exist in exactly one place; this was the first time a convenience asked to move in with them. The design that survived the red-team gives the convenience something else entirely: a **clone** of the operator's brain - the repos, the tooling, the logins, furnished fresh. Never the keys.
<!-- DIAGRAM: browser + phone authentik proxy (every path 302s to login, no carve-outs) code-server pod [own volume: cloned repos, own git key, own logins NO age key] egress only DNS + gitea SSH + 443; the bastion (age key, admin kubeconfigs, SSH keys) sits outside the frame, unreachable. -->
<!-- DIAGRAM: browser + phone -> authentik proxy (every path 302s to login, no carve-outs) -> code-server pod [own volume: cloned repos, own git key, own logins - NO age key] -> egress only DNS + gitea SSH + 443; the bastion (age key, admin kubeconfigs, SSH keys) sits outside the frame, unreachable. -->
![One guarded door to a pod that holds copies and its own scoped identity; the operator's keys sit outside the frame entirely](/diagrams/clone-not-keys.svg)
## Price a stolen session before you build the surface
The question that shaped this build wasn't "how do we secure it?" It was blunter: assume the worst has already happened the session is fully stolen, an attacker sitting in the IDE as you and write down what they now hold.
The question that shaped this build wasn't "how do we secure it?" It was blunter: assume the worst has already happened - the session is fully stolen, an attacker sitting in the IDE as you, and write down what they now hold.
Against the first sketch, the one that ran on the operator host, the red-team returned three separate critical findings, all of them consequences of proximity to that host's keys. Each would have needed its own mitigation, its own monitoring, its own upkeep. Then the design moved to a disposable pod that inherits nothing from the operator host, and all three findings died in the same moment. Not mitigated *deleted*. There was nothing left to steal along those routes, because nothing had been put there.
Against the first sketch, the one that ran on the operator host, the red-team returned three separate critical findings, all of them consequences of proximity to that host's keys. Each would have needed its own mitigation, its own monitoring, its own upkeep. Then the design moved to a disposable pod that inherits nothing from the operator host, and all three findings died in the same moment. Not mitigated - *deleted*. There was nothing left to steal along those routes, because nothing had been put there.
That is the cheapest security work I have ever done: three critical findings closed by a relocation, before a single line of configuration existed. Blast radius is a design input. Treat it as an audit output and you'll be retrofitting forever.
## Clone everything, inherit nothing
Think of the workbench as a reading room attached to an archive. A reading room is genuinely useful every document laid out on the desks, annotated, workable, open late. But the pages on the desks are photocopies. The originals stay in the vault, and no reader's card opens it.
Think of the workbench as a reading room attached to an archive. A reading room is genuinely useful - every document laid out on the desks, annotated, workable, open late. But the pages on the desks are photocopies. The originals stay in the vault, and no reader's card opens it.
The pod starts empty. On first run I furnished it the way you'd stock a reading room: cloned the repos *into* it, generated a fresh git key *inside* it registered in its own name, revocable on its own and signed in to the daily tooling so those logins persist on the pod's own volume. Nothing is mounted from the operator host. Nothing is inherited. And the one thing deliberately kept out is the master secrets key the key that turns the encrypted blobs in those repos into live fleet credentials. The workbench holds a copy of every document; the pages that matter are ciphertext to it.
The pod starts empty. On first run I furnished it the way you'd stock a reading room: cloned the repos *into* it, generated a fresh git key *inside* it - registered in its own name, revocable on its own, and signed in to the daily tooling so those logins persist on the pod's own volume. Nothing is mounted from the operator host. Nothing is inherited. And the one thing deliberately kept out is the master secrets key - the key that turns the encrypted blobs in those repos into live fleet credentials. The workbench holds a copy of every document; the pages that matter are ciphertext to it.
One call in there deserves honesty, because it was a real trade. That git identity has account-level access: the workbench can push, including to the repo the fleet reconciles from. That is genuine power in a convenience surface, and I granted it with eyes open partly because the door in front of it is layered, and mostly because of what the worst case *is*. A hostile push is loud. It lands in git history, attributed and timestamped, and `git revert` undoes it. Set that against the alternative worst case silent possession of every credential in the estate and "attacker can push to some repos" is a failure you recover from, audit, and learn from. The other one is a failure you rebuild from.
One call in there deserves honesty, because it was a real trade. That git identity has account-level access: the workbench can push, including to the repo the fleet reconciles from. That is genuine power in a convenience surface, and I granted it with eyes open - partly because the door in front of it is layered, and mostly because of what the worst case *is*. A hostile push is loud. It lands in git history, attributed and timestamped, and `git revert` undoes it. Set that against the alternative worst case - silent possession of every credential in the estate, and "attacker can push to some repos" is a failure you recover from, audit, and learn from. The other one is a failure you rebuild from.
## Put everything behind one proven door
The front of the workbench is SSO-only, through the identity provider's proxy and *everything* means everything. The route sends 100% of paths through the proxy, with no carve-outs, because the obvious carve-out is a hole: the terminal speaks over a WebSocket, and a WebSocket path excused from authentication is an unauthenticated shell. So it was proven, not assumed probe the root unauthenticated, probe the terminal's WebSocket, probe the static assets and the health endpoint, and every one of them 302s to the login page.
The front of the workbench is SSO-only, through the identity provider's proxy, and *everything* means everything. The route sends 100% of paths through the proxy, with no carve-outs, because the obvious carve-out is a hole: the terminal speaks over a WebSocket, and a WebSocket path excused from authentication is an unauthenticated shell. So it was proven, not assumed - probe the root unauthenticated, probe the terminal's WebSocket, probe the static assets and the health endpoint, and every one of them 302s to the login page.
Behind the SSO sits the application's own password, injected from the secrets manager rather than living in any repo. Belt behind braces: a bug in the proxy still lands on a locked application.
Under both sits a default-deny network policy. The pod can reach the git server, DNS, and its API endpoints over 443 nothing else. That floor is real enough to bite: the git server's own policy had to grant the workbench an ingress seat by name, and the very first clone hung until it did. The hang *was* the policy working.
Under both sits a default-deny network policy. The pod can reach the git server, DNS, and its API endpoints over 443 - nothing else. That floor is real enough to bite: the git server's own policy had to grant the workbench an ingress seat by name, and the very first clone hung until it did. The hang *was* the policy working.
Just as important are the doors that don't exist. A direct LAN bind and a separate mesh-VPN serving path were both sketched, and both cut the one guarded door already serves the couch and the phone alike. Every additional entrance is an additional thing to prove, forever.
Just as important are the doors that don't exist. A direct LAN bind and a separate mesh-VPN serving path were both sketched, and both cut - the one guarded door already serves the couch and the phone alike. Every additional entrance is an additional thing to prove, forever.
## The principle
Every new convenience surface a browser IDE, a remote dashboard, a sync agent should be designed backwards from one number: what a *total* compromise of it yields. Compute that before the surface exists, while the answer is still a design choice. Then engineer it down: clones instead of originals, an identity scoped to the surface and revocable without ceremony, doors layered so no single failure is enough. Only when the worst case is something you could live through do you get to enjoy the convenience and then you get to enjoy it completely, from any couch you like.
Every new convenience surface - a browser IDE, a remote dashboard, a sync agent - should be designed backwards from one number: what a *total* compromise of it yields. Compute that before the surface exists, while the answer is still a design choice. Then engineer it down: clones instead of originals, an identity scoped to the surface and revocable without ceremony, doors layered so no single failure is enough. Only when the worst case is something you could live through do you get to enjoy the convenience, and then you get to enjoy it completely, from any couch you like.
A reading room earns its comfort by holding photocopies. You can deface every page on every desk and the archive survives because the originals were never in the room.
A reading room earns its comfort by holding photocopies. You can deface every page on every desk and the archive survives, because the originals were never in the room.
*Live at code.bztmon.org: a code-server pod on one Talos node, its home a 20Gi volume of cloned repos, its own git key and its own logins behind an Authentik proxy that 302s every path, WebSockets included, to login, above a default-deny network policy. The fleet's age key has never entered it.*
*Live at code.bztmon.org: a code-server pod on one Talos node, its home a 20Gi volume of cloned repos, its own git key and its own logins - behind an Authentik proxy that 302s every path, WebSockets included, to login, above a default-deny network policy. The fleet's age key has never entered it.*
+1 -1
View File
@@ -1,7 +1,7 @@
---
title: "Draft: notes on air-gapped registry mirroring"
date: 2026-06-17
summary: "Work in progress this draft should never appear in the production build or the RSS feed."
summary: "Work in progress - this draft should never appear in the production build or the RSS feed."
tags: ["draft", "registry"]
draft: true
---
@@ -9,9 +9,9 @@ heroAlt: "A GPU glows behind a sealed checkpoint gate while a waiting pod-orb is
---
The most common way a GPU workload fails at the edge isn't the model, the driver, or the network.
It's timing. Kubernetes is eager it will happily schedule your inference pod the moment a node
It's timing. Kubernetes is eager - it will happily schedule your inference pod the moment a node
reports `Ready`, which is often *before* the NVIDIA device plugin has advertised `nvidia.com/gpu`.
The pod starts, can't see a GPU, crash-loops and now your rollout is poisoned across the fleet,
The pod starts, can't see a GPU, crash-loops, and now your rollout is poisoned across the fleet,
on boxes nobody is standing next to.
Node-`Ready` answers the wrong question. It says the kubelet is up. It says nothing about whether
@@ -20,7 +20,7 @@ make GPU readiness explicit, and gate on that.
## Gate the schedule, then gate the start
The first gate is free a resource request. A pod that *requests* a GPU won't schedule until the
The first gate is free - a resource request. A pod that *requests* a GPU won't schedule until the
device plugin advertises capacity:
```yaml
@@ -30,7 +30,7 @@ resources:
```
That handles the common case. But on a single-GPU edge node recovering from a power cut, there's a
window where the plugin has advertised the device and the driver is still finding its feet and
window where the plugin has advertised the device and the driver is still finding its feet - and
you don't want an expensive model load to be the thing that discovers it. So the second gate is an
init container that blocks until the device is demonstrably real, and fails loudly if it never is:
@@ -49,20 +49,19 @@ exit 1
```
Two gates, two failure modes closed: the scheduler can't place the pod before capacity exists, and
the workload can't start before the hardware answers. Note the bound an init gate that waits
the workload can't start before the hardware answers. Note the bound - an init gate that waits
forever isn't a gate, it's a hang. Two and a half minutes, then fail loud and let the platform
retry. Fail-closed, never fail-quiet.
## Why this is the win
Once readiness is gated, the entire class of "pod started before the GPU" failures disappears
and it disappears *the same way on every node*. That consistency is the real prize at the edge.
Once readiness is gated, the entire class of "pod started before the GPU" failures disappears, and it disappears *the same way on every node*. That consistency is the real prize at the edge.
A fix that requires a human to notice, shell in, and nurse a bad rollout doesn't scale past the
first dozen sites; a gate that makes every node converge identically after every reboot does.
## The principle
At the edge, **design the dependency don't hope for it**. Anything your workload cannot run
At the edge, **design the dependency - don't hope for it**. Anything your workload cannot run
without deserves an explicit, bounded, fail-loud gate between it and the scheduler's optimism.
The GPU is just the first dependency worth naming; egress paths and model artifacts are next,
and they want the same treatment.
+18 -22
View File
@@ -1,63 +1,59 @@
---
title: "The best migrations delete more than they install"
date: 2026-08-02
summary: "Nobody chose our network stack it accreted one default at a time, and every component was one more thing to version, patch, debug and alert on. Landing on Cilium wasn't an adoption; it was a demolition schedule, and the stack left standing is smaller than the one we started with."
summary: "Nobody chose our network stack - it accreted one default at a time, and every component was one more thing to version, patch, debug and alert on. Landing on Cilium wasn't an adoption; it was a demolition schedule, and the stack left standing is smaller than the one we started with."
tags: ["cilium", "kubernetes", "networking", "architecture", "homelab"]
draft: false
hero: "/blog/landing-on-cilium.webp"
heroAlt: "An accreted house of circuit-traced extensions dissolves into particles, leaving one clean glowing frame standing"
---
Ask why the fleet ran flannel and there was never an answer worth writing down. Nobody chose it. Talos bundles flannel, so flannel ran. Kubernetes ships kube-proxy, so kube-proxy ran. LoadBalancer services need something on the LAN to answer for their addresses, so MetalLB moved in. HTTP needed terminating, so ingress-nginx went up, and later Traefik replaced it on one cluster, Traefik still answers on ingress-nginx's old IP, the fossil of a hop nobody remembers deciding.
Ask why the fleet ran flannel and there was never an answer worth writing down. Nobody chose it. Talos bundles flannel, so flannel ran. Kubernetes ships kube-proxy, so kube-proxy ran. LoadBalancer services need something on the LAN to answer for their addresses, so MetalLB moved in. HTTP needed terminating, so ingress-nginx went up, and later Traefik replaced it - on one cluster, Traefik still answers on ingress-nginx's old IP, the fossil of a hop nobody remembers deciding.
That's how most platform stacks are built. Not designed accreted, the way an old house grows. A lean-to here, a carport there, each extension the quickest answer to the question of the day. And every extension arrives with its own roofline: a version to pin, a changelog to read, failure modes to learn, alerts to write.
That's how most platform stacks are built. Not designed - accreted, the way an old house grows. A lean-to here, a carport there, each extension the quickest answer to the question of the day. And every extension arrives with its own roofline: a version to pin, a changelog to read, failure modes to learn, alerts to write.
This week the fleet finished landing on Cilium, and the honest record of that landing is the point of this post. It wasn't a product evaluation. It wasn't an adoption. It was a demolition schedule and the stack left standing is smaller than the one we started with.
This week the fleet finished landing on Cilium, and the honest record of that landing is the point of this post. It wasn't a product evaluation. It wasn't an adoption. It was a demolition schedule, and the stack left standing is smaller than the one we started with.
<!-- DIAGRAM: before/after columns on one cluster. Left (before): five separately versioned boxes flannel (CNI), kube-proxy (service path), MetalLB (VIP announcement), Traefik/ingress-nginx (HTTP), and a dashed "NetworkPolicies authored, unenforced" box each with its own version tag. Right (after): a single Cilium box carrying CNI + eBPF service path + LB-IPAM/L2 announcements + Gateway API + enforced policy + Hubble flows, shipped inside the Talos machine document. The transition arrows are strike-throughs (deletions), not installs. -->
![Before and after: five separately versioned components collapse into one engine by deletion](/diagrams/landing-on-cilium.svg)
<!-- DIAGRAM: before/after columns on one cluster. Left (before): five separately versioned boxes - flannel (CNI), kube-proxy (service path), MetalLB (VIP announcement), Traefik/ingress-nginx (HTTP), and a dashed "NetworkPolicies - authored, unenforced" box - each with its own version tag. Right (after): a single Cilium box carrying CNI + eBPF service path + LB-IPAM/L2 announcements + Gateway API + enforced policy + Hubble flows, shipped inside the Talos machine document. The transition arrows are strike-throughs (deletions), not installs. -->
![Before and after: five separately versioned components collapse into one engine - by deletion](/diagrams/landing-on-cilium.svg)
## Count the rooflines, not the features
The starting shape, across a fleet of single-node Talos clusters: flannel carrying pod traffic, kube-proxy rewriting every service address, MetalLB answering ARP for the LoadBalancer VIPs, Traefik terminating HTTP. Plus a set of network policies that were authored, committed and completely inert flannel can't enforce them, so the plans existed and nothing was ever built to them.
The starting shape, across a fleet of single-node Talos clusters: flannel carrying pod traffic, kube-proxy rewriting every service address, MetalLB answering ARP for the LoadBalancer VIPs, Traefik terminating HTTP. Plus a set of network policies that were authored, committed and completely inert - flannel can't enforce them, so the plans existed and nothing was ever built to them.
Four separate projects for four concerns, and a fifth concern going unmet. Each project a chart or a DaemonSet with a version, a CVE feed, an upgrade cadence, and its own distinct way of ruining an evening. None of them wrong individually. All of them defaults and a default is a decision someone else made, for a fleet they've never seen.
Four separate projects for four concerns, and a fifth concern going unmet. Each project a chart or a DaemonSet with a version, a CVE feed, an upgrade cadence, and its own distinct way of ruining an evening. None of them wrong individually. All of them defaults, and a default is a decision someone else made, for a fleet they've never seen.
What started the demolition wasn't "Cilium is better than flannel". It was noticing that one engine could carry all five concerns the pod network, the service path, the VIP announcements, the policy enforcement, the flow visibility and that one cluster in the fleet already ran that engine, a quiet standing proof. Four rooflines could become one.
What started the demolition wasn't "Cilium is better than flannel". It was noticing that one engine could carry all five concerns - the pod network, the service path, the VIP announcements, the policy enforcement, the flow visibility, and that one cluster in the fleet already ran that engine, a quiet standing proof. Four rooflines could become one.
## Swap the frame under an occupied house
Stage one, June: replace the CNI in place, on live clusters, no rebuilds. Per cluster the sequence is short. Set `cni: none` in the machine configuration, one reboot, install Cilium 1.19 with a pinned set of values, delete the flannel DaemonSet, then restart every non-hostNetwork pod so it leaves flannel's address management for Cilium's. Order the clusters lowest blast radius first; the control-plane hub goes last, once the procedure has become boring.
Two disciplines mattered more than the sequence. First, confirm the node rebooted at all the apply returns immediately and the API often answers before the machine has cycled, so trust the uptime counter, not reachability. Second, those inert policies stop being inert the moment Cilium arrives: lines on a plan under flannel, enforced at the instant of cutover. Audit every allow-list *before* migrating the cluster that holds it.
Two disciplines mattered more than the sequence. First, confirm the node rebooted at all - the apply returns immediately and the API often answers before the machine has cycled, so trust the uptime counter, not reachability. Second, those inert policies stop being inert the moment Cilium arrives: lines on a plan under flannel, enforced at the instant of cutover. Audit every allow-list *before* migrating the cluster that holds it.
Four live clusters converted in a day. Zero rebuilds, zero data loss, and the fleet verified green afterwards 98 of 98 GitOps applications healthy. Then stage two, the reason enforcement mattered at all: a default-deny floor in every namespace, rolled fleet-wide with the observe-first method that has [its own post](/blog/observe-first-deny-second) watch the real flows, then deny everything else ([the why lives here](/blog/trust-nothing)).
Four live clusters converted in a day. Zero rebuilds, zero data loss, and the fleet verified green afterwards - 98 of 98 GitOps applications healthy. Then stage two, the reason enforcement mattered at all: a default-deny floor in every namespace, rolled fleet-wide with the observe-first method that has [its own post](/blog/observe-first-deny-second) - watch the real flows, then deny everything else ([the why lives here](/blog/trust-nothing)).
## Then pull the extensions down
Stage three, July finished this week. With Cilium carrying pod traffic everywhere, the remaining structures came down one by one.
Stage three, July - finished this week. With Cilium carrying pod traffic everywhere, the remaining structures came down one by one.
kube-proxy went first. Cilium's eBPF datapath took over service routing entirely, reaching the Kubernetes API through Talos's KubePrism endpoint on `localhost:7445`, after which the kube-proxy DaemonSet was deleted outright. Then MetalLB: Cilium's LB-IPAM handed out the same VIPs from the same pools, L2 announcements answered the same ARP queries, and every service kept its address through the swap. One reboot per node. On the media node, Cilium's Gateway API absorbed the ingress role as well one more structure gone. MetalLB is now extinct across the fleet. Not deprecated, not legacy-mode: extinct.
kube-proxy went first. Cilium's eBPF datapath took over service routing entirely, reaching the Kubernetes API through Talos's KubePrism endpoint on `localhost:7445`, after which the kube-proxy DaemonSet was deleted outright. Then MetalLB: Cilium's LB-IPAM handed out the same VIPs from the same pools, L2 announcements answered the same ARP queries, and every service kept its address through the swap. One reboot per node. On the media node, Cilium's Gateway API absorbed the ingress role as well - one more structure gone. MetalLB is now extinct across the fleet. Not deprecated, not legacy-mode: extinct.
The last demolition was managerial. The CNI no longer belongs to Helm at all it ships as a Talos inline manifest, which means the network lives inside the machine's own configuration document. A rebuilt node comes up with its network the way it comes up with its kernel: because the document says so. Zero Helm release secrets remain anywhere in the fleet.
The last demolition was managerial. The CNI no longer belongs to Helm at all - it ships as a Talos inline manifest, which means the network lives inside the machine's own configuration document. A rebuilt node comes up with its network the way it comes up with its kernel: because the document says so. Zero Helm release secrets remain anywhere in the fleet.
## Two structures fought back
There is always a price, and this one came due twice.
The first trap is the nastiest kind of failure: the change that succeeds and does nothing. Deleting Helm's release record orphans the resources, but it does *not* release Helm's per-field ownership inside them. So the flip to the eBPF datapath applied cleanly, the node rebooted on schedule and `KubeProxyReplacement` still read `False`, because a config field Helm still owned silently refused the new manager's apply. No error. Nothing in the diff. It fired on every single cluster we flipped, so it's now a standard step rather than a contingency: steal the whole manifest's ownership server-side
```
The first trap is the nastiest kind of failure: the change that succeeds and does nothing. Deleting Helm's release record orphans the resources, but it does *not* release Helm's per-field ownership inside them. So the flip to the eBPF datapath applied cleanly, the node rebooted on schedule, and `KubeProxyReplacement` still read `False`, because a config field Helm still owned silently refused the new manager's apply. No error. Nothing in the diff. It fired on every single cluster we flipped, so it's now a standard step rather than a contingency: steal the whole manifest's ownership server-side - ```
kubectl apply --server-side --force-conflicts --field-manager=talos -f cilium-rendered.yaml
```
``` - then verify the config changed before believing anything else.
— then verify the config changed before believing anything else.
The second was musical chairs with the VIPs. Mid-swap on one cluster, a service without a pinned address grabbed another service's freed IP the instant the old announcer released it — the LLM backend ended up sitting on the ingress controller's address. The fix is an ordering rule: pins first. Land the address pins while the old announcer is still alive, confirm they hold, and only then retire it. Two commits, in that order, every time.
The second was musical chairs with the VIPs. Mid-swap on one cluster, a service without a pinned address grabbed another service's freed IP the instant the old announcer released it - the LLM backend ended up sitting on the ingress controller's address. The fix is an ordering rule: pins first. Land the address pins while the old announcer is still alive, confirm they hold, and only then retire it. Two commits, in that order, every time.
## The principle
Consolidation is usually sold on the new tool's feature list. That's the wrong ledger. The value of landing on one engine wasn't what Cilium added it was what the landing let us remove: a CNI, a service proxy, a VIP announcer, and on one node an ingress controller, each of which had been a version to pin, a feed to watch, an alert to tune, a way to be paged. Policy enforcement and flow observability never needed projects of their own; they came up with the same engine, side effects of the consolidation rather than line items on it.
Consolidation is usually sold on the new tool's feature list. That's the wrong ledger. The value of landing on one engine wasn't what Cilium added - it was what the landing let us remove: a CNI, a service proxy, a VIP announcer, and on one node an ingress controller, each of which had been a version to pin, a feed to watch, an alert to tune, a way to be paged. Policy enforcement and flow observability never needed projects of their own; they came up with the same engine, side effects of the consolidation rather than line items on it.
Every structure you tear down is one that can never fall on you. Judge the next migration by its demolition list.
+18 -18
View File
@@ -1,7 +1,7 @@
---
title: "My servers don't have SSH, and that's the feature"
date: 2026-07-14
summary: "Every box in this fleet runs an OS with no shell, no package manager, and no SSH daemon the entire machine is an API with a declarative config. It sounds like giving up control. It's the opposite: you can't drift what you can't touch."
summary: "Every box in this fleet runs an OS with no shell, no package manager, and no SSH daemon - the entire machine is an API with a declarative config. It sounds like giving up control. It's the opposite: you can't drift what you can't touch."
tags: ["talos", "kubernetes", "immutable-infrastructure", "security", "homelab"]
draft: false
hero: "/blog/no-ssh.webp"
@@ -20,61 +20,61 @@ realise it's the point.
Every hand-run command on a traditional box is an unrecorded change: a config edited to test something, a
package installed during an incident, a daemon restarted with a flag nobody wrote down. None of it is in
git. All of it is load-bearing by next month. Configuration drift isn't a failure of discipline it's the
git. All of it is load-bearing by next month. Configuration drift isn't a failure of discipline - it's the
*inevitable* product of machines that accept hands.
Talos removes the hands. The entire machine disks, network interfaces pinned by MAC, kernel modules,
registry mirrors, even the manifests the cluster boots with is one declarative document, applied through
Talos removes the hands. The entire machine - disks, network interfaces pinned by MAC, kernel modules,
registry mirrors, even the manifests the cluster boots with - is one declarative document, applied through
the API, versioned in git. There's no side door through which an undocumented change can arrive, because
there's no door. The machine config isn't documentation *of* the machine. It **is** the machine.
<!-- DIAGRAM: left a traditional server as a house with many doors (ssh, console, package manager), arrows of drift entering; right a sealed Talos block with one structured API port, a signed config document flowing in -->
<!-- DIAGRAM: left - a traditional server as a house with many doors (ssh, console, package manager), arrows of drift entering; right - a sealed Talos block with one structured API port, a signed config document flowing in -->
![Many doors and drift, versus one API and a document](/diagrams/no-ssh.svg)
## Operating through the keyhole
Day-two work changes shape. Logs, service status, process lists, even packet capture arrive through typed
API calls, not a terminal session and every mutation is an *apply*: edit the document, run a dry-run diff
API calls, not a terminal session, and every mutation is an *apply*: edit the document, run a dry-run diff
that says precisely what will change and whether it costs a reboot, then commit it. The dry-run gate is the
quiet superpower on a machine you can't shell into, "what exactly will this do?" stops being a guess and
quiet superpower - on a machine you can't shell into, "what exactly will this do?" stops being a guess and
becomes an answer the API owes you *before* anything moves.
This week put that shape under load. Three of the six nodes had their network datapath swapped kube-proxy
out, Cilium's eBPF replacement in and their registry credentials rotated, one reboot each, entirely
This week put that shape under load. Three of the six nodes had their network datapath swapped - kube-proxy
out, Cilium's eBPF replacement in, and their registry credentials rotated, one reboot each, entirely
through the API. Nobody logged into anything, because there is nothing to log into. The credentials never
touched a live machine either: they're substituted into the document at render time from an encrypted file
and arrive as configuration, not keystrokes there is no terminal to mistype them into.
and arrive as configuration, not keystrokes - there is no terminal to mistype them into.
Two habits from that work show what keyhole operation feels like. After an apply that costs a reboot, the
API answers again *before* the machine has cycled the door reopening proves nothing. So you read
API answers again *before* the machine has cycled - the door reopening proves nothing. So you read
`/proc/uptime` through the API and demand a number seconds old; the check has caught a node still showing
413,223 seconds of uptime with the apply long returned. You don't trust the door. You read the building's
own clock through the keyhole. And before calling a node good, you have it pull a container image through
that same API the machine exercises its new registry credential end-to-end while you watch from outside.
that same API - the machine exercises its new registry credential end-to-end while you watch from outside.
When something's genuinely wrong, the recovery isn't archaeology on a mutated filesystem it's
When something's genuinely wrong, the recovery isn't archaeology on a mutated filesystem - it's
reconciliation: re-apply the known-good document, or in the worst case rebuild the node from it in minutes.
The machine has no state worth rescuing *because nothing was ever hand-placed on it*. Pets die of unknown
illnesses; documents get reprinted.
## The honest trade
The console is gone as a *control* surface, not as a *truth* surface and that distinction still bites. A
The console is gone as a *control* surface, not as a *truth* surface, and that distinction still bites. A
node once dropped off the network in a way that looked, from every remote signal, like a total lockup. The
physical screen told a different story: the box was alive and healthy, only its network path had wedged.
The lesson isn't "you need a shell" a shell would have shown the same thing more slowly. It's that
The lesson isn't "you need a shell" - a shell would have shown the same thing more slowly. It's that
firsthand evidence still outranks remote inference, and an API-only fleet needs its operators to remember
the difference on the day it matters. The same lesson came back this week when a node went dark at layer 2:
the fix was a screwdriver, not a session no SSH daemon answers on a dead NIC either.
the fix was a screwdriver, not a session - no SSH daemon answers on a dead NIC either.
## The principle
Control isn't the ability to touch a machine it's the guarantee that the machine matches a document you
Control isn't the ability to touch a machine - it's the guarantee that the machine matches a document you
trust. Every interactive door a server offers is a place where reality and the record can quietly diverge,
and everything that makes fleets miserable lives in that gap. Seal the doors, apply the document, keep the
diff. The machine you can't log into is the only one whose state you truly know.
*Live across a six-node Talos fleet: machine configs generated and versioned in git, applied over mTLS with
dry-run gates, nodes rebuilt from documents when hardware moves. Three of the six had their network datapath
swapped and their registry credentials rotated this week one reboot each, entirely through the API. No SSH
swapped and their registry credentials rotated this week - one reboot each, entirely through the API. No SSH
key to any of them exists, because there is nothing for it to open.*
+23 -23
View File
@@ -1,7 +1,7 @@
---
title: "Observe first, deny second"
date: 2026-07-14
summary: "Everyone writes network policy from the architecture diagram, and the diagram is always wrong. The only allowlist that survives contact with production is one written from the flows you actually watched applied out-of-band, proven enforcing, and only then handed to GitOps."
summary: "Everyone writes network policy from the architecture diagram, and the diagram is always wrong. The only allowlist that survives contact with production is one written from the flows you actually watched - applied out-of-band, proven enforcing, and only then handed to GitOps."
tags: ["security", "networking", "cilium", "hubble", "kubernetes", "gitops"]
draft: false
hero: "/blog/observe-first-deny-second.webp"
@@ -10,10 +10,10 @@ heroAlt: "A dark reserve at night, instrument beams tracing animal paths before
Here's the uncomfortable default: when it's time to lock a namespace down, almost everyone opens the
architecture diagram and starts writing allow rules from it. This talks to that, that talks to the database,
done. The diagram is confident, tidy and wrong. It omits the probe traffic, the DNS hop, the controller's
done. The diagram is confident, tidy, and wrong. It omits the probe traffic, the DNS hop, the controller's
back-channel to the API server, the one call some library makes that nobody documented.
Policy written from a diagram *feels* like engineering. It's fiction with enforcement attached and
Policy written from a diagram *feels* like engineering. It's fiction with enforcement attached - and
enforcement doesn't care that the fiction was well-intentioned. It drops the flow anyway, in production,
at whatever hour the flow next happens.
@@ -21,15 +21,15 @@ I made the case for the default-deny floor itself in [the trust-nothing post](/b
the other half: how you roll it across a live fleet, namespace by namespace, without an outage. The inversion is
simple. Don't fence first and see what starves. **Survey the reserve, then build the fence.**
<!-- DIAGRAM: the loop Hubble observe write allowlist from evidence apply out-of-band verify enforcing + zero drops commit to git -->
![The observe deny verify commit loop](/diagrams/observe-first-deny-second.svg)
<!-- DIAGRAM: the loop - Hubble observe -> write allowlist from evidence -> apply out-of-band -> verify enforcing + zero drops -> commit to git -->
![The observe -> deny -> verify -> commit loop](/diagrams/observe-first-deny-second.svg)
## Survey before you fence
A ranger doesn't fence a reserve from a map. They track the actual animals for a season where they drink,
which paths they take at night because the fence has to leave every real path open and close everything else.
A ranger doesn't fence a reserve from a map. They track the actual animals for a season - where they drink,
which paths they take at night, because the fence has to leave every real path open and close everything else.
Hubble is that season of tracking. Before a single policy exists, watch the namespace's real flows every
Hubble is that season of tracking. Before a single policy exists, watch the namespace's real flows - every
connection, source and destination identity, port. That record *is* the allowlist. Not the docs, not your
memory of the architecture, not the diagram. Every rule you write should point at a flow you watched happen.
Start with the lowest-risk namespaces and leave the control plane for last: mis-fence a leaf app and one thing
@@ -37,9 +37,9 @@ breaks; mis-fence the plane that runs deploys and everything does.
## Build the fence where you can tear it down
Apply the policy trio the default-deny plus its allows **out-of-band** with `kubectl`, not through git.
Apply the policy trio (the default-deny plus its allows) **out-of-band** with `kubectl`, not through git.
This is deliberate, and it's the step people skip. The GitOps reconciler doesn't know the policy exists, so
it can't fight you over it, and rollback is one `kubectl delete` instant, no commit, no sync wait. Push the
it can't fight you over it, and rollback is one `kubectl delete`: instant, no commit, no sync wait. Push the
policy through git first and you've inverted your own safety: selfHeal means the reconciler restores whatever
you delete, in seconds, while you stand there believing you rolled back.
@@ -50,33 +50,33 @@ gates.
Two verifications, and both are mandatory because each one lies without the other.
First, prove the policy enforces at all. On Cilium 1.19 a policy can be **accepted but inert** it passes a
First, prove the policy enforces at all. On Cilium 1.19 a policy can be **accepted but inert** - it passes a
server-side dry-run, sits in the cluster looking correct, and enforces nothing. A policy that looks live and
isn't is worse than no policy: it changes what you believe without changing what the network does. Read the
live object's status conditions and demand `Valid=True`, then watch a forbidden connection get dropped. "It
applied cleanly" and "it's enforcing" are different sentences.
Second, prove nothing legitimate is bleeding. Back to Hubble zero drops on real flows, and the real consumer
Second, prove nothing legitimate is bleeding. Back to Hubble - zero drops on real flows, and the real consumer
path exercised end-to-end, not just a curl from your own shell.
Only when both hold does the policy go to git. The reconciler adopts it, out-of-band ends, and the rollback
story changes shape: from here, undo means `git revert` because selfHeal now defends the policy as hard as
story changes shape: from here, undo means `git revert`, because selfHeal now defends the policy as hard as
it would have fought your rollback.
The sternest test so far wasn't a rollout at all: the fleet's datapath was swapped out underneath the
policies kube-proxy replaced by Cilium's eBPF kube-proxy-replacement on the last three clusters, one reboot
each and afterwards every namespace's floor re-verified enforcing: 30/30, 32/32, 27/27 `Valid=True`, zero
policies - kube-proxy replaced by Cilium's eBPF kube-proxy-replacement on the last three clusters, one reboot
each, and afterwards every namespace's floor re-verified enforcing: 30/30, 32/32, 27/27 `Valid=True`, zero
legitimate drops. The fence held while the ground under it was replaced.
## The paths nobody draws
Three flows exist in every namespace and appear on no diagram. Omit any one and you pay:
- **DNS allowed by label, never by IP.** Pin the resolver's ClusterIP into a rule and it never matches at
all the address is rewritten to a backend pod before policy judges the flow. Select `k8s-app: kube-dns`
- **DNS: allowed by label, never by IP.** Pin the resolver's ClusterIP into a rule and it never matches at
all: the address is rewritten to a backend pod before policy judges the flow. Select `k8s-app: kube-dns`
and it holds forever.
- **The kubelet's probes.** Health checks arrive from the **host** entity on *every* workload. Forget the
allow and your pods go NotReady the moment the deny lands the fence starving the animals it was meant
allow and your pods go NotReady the moment the deny lands, the fence starving the animals it was meant
to protect.
- **API-server egress.** Every controller and operator talks to the Kubernetes API constantly. A controller
that "manages X" usually manages it *through* the API server, so this one allow often covers its whole
@@ -84,13 +84,13 @@ Three flows exist in every namespace and appear on no diagram. Omit any one and
## The principle
The method generalises past network policy. Any control that turns assumption into enforcement firewall,
RBAC, admission is only as honest as the evidence it was written from. Observation first, out-of-band while
The method generalises past network policy. Any control that turns assumption into enforcement - firewall,
RBAC, admission - is only as honest as the evidence it was written from. Observation first, out-of-band while
unproven, permanent only after it's earned it.
Policy written from observation is engineering. Policy written from a diagram is fiction and production is
Policy written from observation is engineering. Policy written from a diagram is fiction, and production is
where fiction gets fact-checked.
*Rolled out across all six Talos clusters on Cilium 1.19 every namespace surveyed with Hubble before its
*Rolled out across all six Talos clusters on Cilium 1.19 - every namespace surveyed with Hubble before its
deny landed, the control plane last, zero outages; the floor re-verified enforcing after this week's
kube-proxyeBPF datapath swap.*
kube-proxy->eBPF datapath swap.*
+17 -17
View File
@@ -1,58 +1,58 @@
---
title: "SSO is for people, not for machines"
date: 2026-08-02
summary: "When an app grows a human face on an interface machines also depend on, one auth layer over everything either breaks the machines or quietly weakens the human door to let them through. The pattern that holds: a single identity-provider proxy in front of every human door, while machine paths keep their own scoped credentials route by audience, not by app."
summary: "When an app grows a human face on an interface machines also depend on, one auth layer over everything either breaks the machines or quietly weakens the human door to let them through. The pattern that holds: a single identity-provider proxy in front of every human door, while machine paths keep their own scoped credentials - route by audience, not by app."
tags: ["security", "sso", "authentik", "kubernetes", "homelab"]
draft: false
hero: "/blog/one-doorman-many-doors.webp"
heroAlt: "A glowing reception desk in a dark circuit-lined lobby signs streams of visitor light through many doors, while courier light-trails slip through keyed loading docks around the building's edge"
---
Sooner or later, a machine-facing service grows a human face. The container registry every node in the fleet pulls from ships a web UI. The development environment becomes a browser tab with a live terminal in it. Now there are people arriving at hostnames that machines also depend on and the tempting move is one auth layer over the whole thing. Put the entire host behind single sign-on. One rule, no exceptions, done.
Sooner or later, a machine-facing service grows a human face. The container registry every node in the fleet pulls from ships a web UI. The development environment becomes a browser tab with a live terminal in it. Now there are people arriving at hostnames that machines also depend on, and the tempting move is one auth layer over the whole thing. Put the entire host behind single sign-on. One rule, no exceptions, done.
Do that to a registry and the cluster stops pulling images the same afternoon, because a container runtime cannot answer an interactive login page. The failure after that one is quieter and worse: you see the machines breaking, so you loosen the layer to let them through an exception here, an anonymous path there until the human door is weaker *because* the machines needed in.
Do that to a registry and the cluster stops pulling images the same afternoon, because a container runtime cannot answer an interactive login page. The failure after that one is quieter and worse: you see the machines breaking, so you loosen the layer to let them through - an exception here, an anonymous path there - until the human door is weaker *because* the machines needed in.
The answer isn't a cleverer single layer. It's a split: **route by audience, not by app**. People get one doorman. Machines get keyed doors of their own, nowhere near the desk.
<!-- DIAGRAM: One hostname, two audiences. Left: human browsers converge on a single identity-provider proxy outpost (the doorman), which fronts multiple app doors registry web UI and browser IDE each app a provider bound to the SAME outpost. Right: machine clients (node container runtimes, CI) hit the registry's /v2 path directly, split off at the gateway, presenting their own scoped read-only credential never touching the SSO layer. Show the gateway making the split: /v2 registry direct, catch-all outpost. Cyan #0c8fce human path, magenta #c026d3 machine path, violet #7c3aed app boxes, on the light card. -->
![One host, two doors the gateway splits by audience: browsers route through the identity-provider outpost, while the registry's /v2 machine path goes direct with its own scoped credential](/diagrams/one-doorman-many-doors.svg)
<!-- DIAGRAM: One hostname, two audiences. Left: human browsers converge on a single identity-provider proxy outpost (the doorman), which fronts multiple app doors - registry web UI and browser IDE - each app a provider bound to the SAME outpost. Right: machine clients (node container runtimes, CI) hit the registry's /v2 path directly, split off at the gateway, presenting their own scoped read-only credential - never touching the SSO layer. Show the gateway making the split: /v2 -> registry direct, catch-all -> outpost. Cyan #0c8fce human path, magenta #c026d3 machine path, violet #7c3aed app boxes, on the light card. -->
![One host, two doors - the gateway splits by audience: browsers route through the identity-provider outpost, while the registry's /v2 machine path goes direct with its own scoped credential](/diagrams/one-doorman-many-doors.svg)
## A doorman is for doors that can't answer for themselves
The house rule comes before anything else: when an app speaks OIDC natively, wire it natively. The app authenticates its own users against the identity provider and stays directly reachable which matters, because its existing machine clients (a credential vault's mobile app, a git CLI pushing over HTTPS) keep working untouched. Native wiring routes by audience for free.
The house rule comes before anything else: when an app speaks OIDC natively, wire it natively. The app authenticates its own users against the identity provider and stays directly reachable, which matters, because its existing machine clients (a credential vault's mobile app, a git CLI pushing over HTTPS) keep working untouched. Native wiring routes by audience for free.
A proxy outpost an authenticating reverse proxy operated by the identity provider is for the remainder: apps with no serious auth story of their own, or surfaces you'd never trust to defend themselves. Two tenants in this fleet fit that bill. A registry web UI, whose built-in auth is a static htpasswd file a fine key for a machine, a poor front door for a person. And a browser IDE an editor in a tab with a real shell behind it, which is about the most consequential thing you can put behind a URL.
A proxy outpost (an authenticating reverse proxy operated by the identity provider) is for the remainder: apps with no serious auth story of their own, or surfaces you'd never trust to defend themselves. Two tenants in this fleet fit that bill. A registry web UI, whose built-in auth is a static htpasswd file: a fine key for a machine, a poor front door for a person. And a browser IDE: an editor in a tab with a real shell behind it, which is about the most consequential thing you can put behind a URL.
## One desk signs in every visitor
Recent versions of the identity provider dropped their embedded proxy, so the outpost is now a deployment you run yourself. That sounds like a tax. It's the opposite, because you only ever run one. A single outpost serves many providers: each new application binds a new provider to the *same* instance. When the browser IDE arrived the second tenant the marginal cost was a provider object, a route pointing its hostname at the outpost, and a matching pair of network policies. No new deployment. App N+1 is a nameplate at the desk, not a second desk.
Recent versions of the identity provider dropped their embedded proxy, so the outpost is now a deployment you run yourself. That sounds like a tax. It's the opposite, because you only ever run one. A single outpost serves many providers: each new application binds a new provider to the *same* instance. When the browser IDE arrived - the second tenant - the marginal cost was a provider object, a route pointing its hostname at the outpost, and a matching pair of network policies. No new deployment. App N+1 is a nameplate at the desk, not a second desk.
Standing up the desk the first time is where the hours went. Three traps, each with a price paid in real time:
- **A provider created by automation isn't a provider created by the UI.** Scripting the provider into existence skipped the OAuth defaults the admin UI sets silently the redirect URIs sat empty, and every successful login dumped the user onto the identity provider's own homepage with a shrug of "no provider url". The login worked; it just went nowhere. Set the defaults explicitly, every time.
- **The outpost has to be told which host the browser sees.** Left alone it knows only its internal service name, and it will happily redirect your browser to an address that exists nowhere outside the cluster. One environment variable carrying the browser-facing hostname closes the loop and it wins over anything set through the API.
- **A provider created by automation isn't a provider created by the UI.** Scripting the provider into existence skipped the OAuth defaults the admin UI sets silently - the redirect URIs sat empty, and every successful login dumped the user onto the identity provider's own homepage with a shrug of "no provider url". The login worked; it just went nowhere. Set the defaults explicitly, every time.
- **The outpost has to be told which host the browser sees.** Left alone it knows only its internal service name, and it will happily redirect your browser to an address that exists nowhere outside the cluster. One environment variable carrying the browser-facing hostname closes the loop, and it wins over anything set through the API.
- **A token with an invisible trailing newline fails like everything else fails.** The outpost's token, pasted into a secret, carried a newline you cannot see. The authorisation header it produces is invalid, and the symptoms are indistinguishable from both traps above. Strip the newline before the secret ever exists.
## Couriers don't queue at reception
The registry is the sharper case, because its real customers were never people. Every node's container runtime pulls images through its API the `/v2` path dozens of times a day. Put the doorman in front of that and every pull in the fleet dies waiting on a login page no runtime can answer.
The registry is the sharper case, because its real customers were never people. Every node's container runtime pulls images through its API - the `/v2` path - dozens of times a day. Put the doorman in front of that and every pull in the fleet dies waiting on a login page no runtime can answer.
So the route splits at the gateway. `/v2` goes straight to the registry. Everything else on the same hostname the UI, the search goes to the outpost and gets the full sign-in. And the direct path isn't left swinging open as the price of working: it's keyed with its own scoped, read-only identity, carried in each node's registry configuration. The couriers don't queue at reception, and reception doesn't prop the dock open for them either.
So the route splits at the gateway. `/v2` goes straight to the registry. Everything else on the same hostname - the UI, the search - goes to the outpost and gets the full sign-in. And the direct path isn't left swinging open as the price of working: it's keyed with its own scoped, read-only identity, carried in each node's registry configuration. The couriers don't queue at reception, and reception doesn't prop the dock open for them either.
One hostname, two audiences, two doors and neither is weakened to accommodate the other.
One hostname, two audiences, two doors, and neither is weakened to accommodate the other.
## Rattle every door before you trust the lock
The proof discipline matters as much as the pattern. The easy check is to load the homepage, watch it bounce to the sign-in page, and call the thing secured. But the homepage is the door nobody breaks in through.
The browser IDE made this concrete. Its whole value is a terminal speaking over a WebSocket and a WebSocket doesn't render a login page. It either connects or it doesn't. Had the route carved that path out for convenience, the front door would be locked while a side corridor ran straight past the desk, exactly where no identity is ever checked. The same goes for static assets, and for the health endpoint.
The browser IDE made this concrete. Its whole value is a terminal speaking over a WebSocket, and a WebSocket doesn't render a login page. It either connects or it doesn't. Had the route carved that path out for convenience, the front door would be locked while a side corridor ran straight past the desk, exactly where no identity is ever checked. The same goes for static assets, and for the health endpoint.
So the route sends every path on the host to the outpost, and the check was empirical: an unauthenticated request to the root, to the terminal WebSocket, to a static asset and to the health endpoint each came back as a redirect to the identity provider. All of them. On a human door, a single carve-out *is* the hole; carve-outs belong only on machine doors, where they get keys of their own. And behind the doorman, the IDE still keeps a lock of its own the desk adds a layer; it doesn't replace the one on the office door.
So the route sends every path on the host to the outpost, and the check was empirical: an unauthenticated request to the root, to the terminal WebSocket, to a static asset and to the health endpoint each came back as a redirect to the identity provider. All of them. On a human door, a single carve-out *is* the hole; carve-outs belong only on machine doors, where they get keys of their own. And behind the doorman, the IDE still keeps a lock of its own - the desk adds a layer; it doesn't replace the one on the office door.
## The principle
Slice authentication by who arrives, not by what serves the request. Humans are one audience interactive, browser-borne, phishable and they get a doorman: one strong flow, every human door in the estate behind the same desk, so door N+1 inherits the whole policy for the cost of a nameplate. Machines are the other audience headless, scoped, incapable of answering a challenge and they get narrow keys to their own doors, which the doorman never learns about.
Slice authentication by who arrives, not by what serves the request. Humans are one audience - interactive, browser-borne, phishable, and they get a doorman: one strong flow, every human door in the estate behind the same desk, so door N+1 inherits the whole policy for the cost of a nameplate. Machines are the other audience - headless, scoped, incapable of answering a challenge, and they get narrow keys to their own doors, which the doorman never learns about.
The one-layer instinct fails because it treats the app as the unit of security. The unit is the audience. Sign the people in at one desk, key the machines at the dock and neither door ever has to weaken to let the other's traffic through.
The one-layer instinct fails because it treats the app as the unit of security. The unit is the audience. Sign the people in at one desk, key the machines at the dock, and neither door ever has to weaken to let the other's traffic through.
*Live in the homelab: one Authentik proxy outpost on the cave cluster fronting the Zot registry UI and a code-server IDE; the registry's `/v2` pull path keyed separately with a read-only identity across six Talos nodes.*
+3 -3
View File
@@ -15,14 +15,14 @@ that's online whether or not anyone's knocking.
There's a better shape: don't open anything. Let the origin dial *out*.
![Outbound-only exposure public traffic hits a Cloudflare edge with WAF and geo rules; the origin holds a single outbound tunnel; nothing is port-forwarded](/diagrams/outbound-exposure.svg?v=2)
![Outbound-only exposure - public traffic hits a Cloudflare edge with WAF and geo rules; the origin holds a single outbound tunnel; nothing is port-forwarded](/diagrams/outbound-exposure.svg?v=2)
## The tunnel dials out
A lightweight connector runs next to the service and opens a persistent **outbound** connection to the edge
network. Public traffic arrives at the edge, and the edge hands it back down that already-open tunnel. The
router has no inbound rule. The origin's public IP is never advertised. Port-scan the home connection and
there's nothing listening because there isn't.
there's nothing listening, because there isn't.
You've inverted the trust direction: instead of the internet reaching *in*, the box reaches *out*.
@@ -30,7 +30,7 @@ You've inverted the trust direction: instead of the internet reaching *in*, the
The wrinkle is DNS. I run **split-horizon**: the internal domain resolves to in-cluster ingress for anyone
on the LAN, and the public domain resolves through the edge for the outside world. Same services, two names,
two answers depending on where you're standing and the internal estate is never reachable through the
two answers depending on where you're standing, and the internal estate is never reachable through the
public path.
This bites in a non-obvious way too: a pod *inside* the cluster can't resolve the public hostname (it gets
+5 -5
View File
@@ -1,18 +1,18 @@
---
title: "A new model is a new hire, not a new file"
date: 2026-08-17
summary: "A model pulled from the hub is not just weights - it ships tokeniser code, an executable chat template, loaders gated by trust_remote_code, sometimes its own runtime, and the ecosystem's default is to run all of it beside your credentials. So every new model starts on probation: a hardware-isolated microVM with its own guest kernel, a default-deny network, zero credentials - and promotion to the standard serving tier only when the observed record earns it."
summary: "A model pulled from the hub is not just weights - it ships tokeniser code, an executable chat template, loaders gated by trust_remote_code, sometimes its own runtime, and the ecosystem's default is to run all of it beside your credentials. So every new model starts on probation: a hardware-isolated microVM with its own guest kernel, a default-deny network, zero credentials, and promotion to the standard serving tier only when the observed record earns it."
tags: ["ai", "security", "kata", "isolation", "homelab"]
draft: false
hero: "/blog/probation-for-models.webp"
heroAlt: "A new arrival's first supervised shift: an android-like figure of light works alone inside a glass observation room on a vast dark facility floor, instruments watching from outside, the busy permanent floor glowing far beyond"
---
Pull a model from the hub and notice what your own head does with it. A directory of large binary files arrives, so it gets filed under *data*. Weights. Tensors. Numbers - and numbers can't run.
Pull a model from the hub and notice what your own head does with it. A directory of large binary files arrives, so it gets filed under *data*. Weights. Tensors. Numbers, and numbers can't run.
Except that isn't what you downloaded. A model artefact ships tokeniser code. It ships a chat template - an executable template the runtime evaluates on every request. It often ships a loader gated by a flag whose name confesses everything, `trust_remote_code=True`, and sometimes it effectively ships its own runtime, pulled as a container image from someone else's registry. The ecosystem's default is to run all of this with cluster-level convenience: the same kernel as everything else you host, the same namespace, a service-account token mounted at the usual path. Treat a fresh model as data and you will, sooner or later, execute a stranger's Python next to your credentials.
I've argued before that untrusted code belongs in a VM, not a namespace; the downloaded model is that argument's most routinely ignored case. So the operating rule in this fleet is a hiring rule, not a file-handling one: **new models start on probation**. You didn't add an asset to storage. You hired a stranger off the internet - and a new hire doesn't get keys to anything on day one.
I've argued before that untrusted code belongs in a VM, not a namespace; the downloaded model is that argument's most routinely ignored case. So the operating rule in this fleet is a hiring rule, not a file-handling one: **new models start on probation**. You didn't add an asset to storage. You hired a stranger off the internet, and a new hire doesn't get keys to anything on day one.
<!-- DIAGRAM: the probation ladder, left to right. ARRIVE: a model artefact (weights + tokeniser code + chat template + loader), pinned by digest with its runtime. PROBATION TIER (live machinery): Sandbox CR -> pod with runtimeClassName kata -> per-pod KVM microVM with its OWN guest kernel, inside a zero-peer default-deny namespace; zero credentials mounted; egress opened only to declared needs; broker tier above holds all real keys. OBSERVE: real workloads through the harness; the record = egress log + behaviour vs declaration. GATE: record matches declaration over time -> PROMOTE to the standard serving tier behind the gateway; unproven -> stays in probation indefinitely (cheap). Highlight the hardware boundary between the container and the node kernel. -->
![The probation ladder: arrive pinned, run supervised in a microVM behind default-deny with zero credentials, build a record, promote only when the record matches the declaration](/diagrams/probation-for-models.svg)
@@ -23,7 +23,7 @@ Consider what you'd normally know about a new starter: references you can call,
Meanwhile the artefact's executable surface is wider than most people's mental picture of it. The older checkpoint formats are pickles, and unpickling is code execution - safetensors exists precisely because "loading the weights" used to mean "running a program". Custom architectures ask for `trust_remote_code`, which imports and runs whatever Python the repository carries, in your process, with your permissions. Even the innocuous chat template is a small program, executed on every request. Little of this is malice; nearly all of it is engineering convenience. But convenience is how a stranger's code ends up running beside a token that can list every secret in the namespace.
You cannot call this candidate's references. The only reference check available is watching it work - which is what a probation period is.
You cannot call this candidate's references. The only reference check available is watching it work, which is what a probation period is.
## Give the stranger a supervised shift
@@ -31,7 +31,7 @@ The probation tier here is not a policy document; it's a namespace, and everythi
A `Sandbox` resource (kubernetes-sigs/agent-sandbox) wraps that pod and owns its lifecycle, and the namespace floor does the supervising. The network is default-deny with zero peers - a fresh sandbox can pull its image and reach nothing else until someone writes down, explicitly, the handful of addresses it's allowed. Admission rejects any pod that tries to start in that namespace without the microVM runtime, so nobody can absent-mindedly schedule a shortcut. And the tier holds no credentials at all, by design: anything a workload legitimately needs is asked for through the broker layer above it, which holds the real keys - the pattern from [an agent should never hold the key it's using](/blog/broker-pattern).
That's a supervised shift in full. Escorted everywhere it goes, nothing in its pockets, every outside contact through a chaperone - and the building stays standing even if the new starter turns out to be hostile.
That's a supervised shift in full. Escorted everywhere it goes, nothing in its pockets, every outside contact through a chaperone, and the building stays standing even if the new starter turns out to be hostile.
## Promote on the record, not the calendar
+13 -13
View File
@@ -1,10 +1,10 @@
---
title: "A backup you haven't restored is a rumour"
date: 2026-07-03
summary: "Backup jobs report success every night for years and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught."
summary: "Backup jobs report success every night for years, and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught."
tags: ["backups", "disaster-recovery", "kubernetes", "restic", "homelab"]
hero: "/blog/restore-or-rumour.webp?v=1"
heroAlt: "A dim vault with a small tray of retrieved data-crystals under an examination lamp the proof is what was brought back OUT, not what sits sealed inside."
heroAlt: "A dim vault with a small tray of retrieved data-crystals under an examination lamp - the proof is what was brought back OUT, not what sits sealed inside."
draft: false
---
@@ -12,33 +12,33 @@ draft: false
Ask an estate whether it has backups and you'll get a confident yes: the job runs nightly, the dashboard is
green, the repository is growing. Ask when anyone last *restored* one and the room goes quiet.
Here's the uncomfortable framing: an unverified backup is not evidence it's a rumour. Someone once said
Here's the uncomfortable framing: an unverified backup is not evidence - it's a rumour. Someone once said
the data was safe, the claim got repeated nightly by a cron job, and everyone chose to believe it. The green
tick proves a process exited zero. It says nothing about whether the bytes at rest can be turned back into a
running application on the worst day of the year.
## The restore drill is the acceptance test
When I consolidated the homelab's backups onto the NAS every cluster's application state streaming into
one repository per cluster the job wasn't "done" when the first snapshot landed. It was done when a
When I consolidated the homelab's backups onto the NAS - every cluster's application state streaming into
one repository per cluster - the job wasn't "done" when the first snapshot landed. It was done when a
scratch pod pulled thousands of files back *out* of the repository, the configs checked intact, and the
services proved recoverable from nothing but the backup and its key. Both clusters. Before the old backup
path was retired, not after.
That's the rule worth writing down: **a backup pipeline ships with its restore drill, the same way code
ships with its tests.** Not a runbook that describes a restore an actual rehearsal, on the real
ships with its tests.** Not a runbook that describes a restore - an actual rehearsal, on the real
repository, with a pass/fail outcome.
<!-- DIAGRAM: nightly snapshot arrows into a repository; a drill loop pulling a snapshot back OUT into a scratch pod with a green check the loop is what's highlighted, not the vault -->
<!-- DIAGRAM: nightly snapshot arrows into a repository; a drill loop pulling a snapshot back OUT into a scratch pod with a green check - the loop is what's highlighted, not the vault -->
![The drill loop: data out of the vault is the proof, not data in](/diagrams/restore-or-rumour.svg?v=1)
## What the drill actually catches
The failures a drill surfaces are precisely the ones the nightly tick can't see. A repository still locked
by a dead process from last week's crashed run every subsequent "successful" backup queued behind a stale
by a dead process from last week's crashed run - every subsequent "successful" backup queued behind a stale
lock. A prune step that had been silently failing. And the one that matters most on the worst day: whether
the *key* is where the recovery plan says it is. Every one of those was invisible from the green-tick view,
and every one is a nothing-burger to fix on a Tuesday afternoon and an incident report if discovered
and every one is a nothing-burger to fix on a Tuesday afternoon, and an incident report if discovered
during a real recovery.
There's a quieter design lesson the drill enforces too: back up **state, not just volumes**. A database dump
@@ -47,7 +47,7 @@ the difference, because the drill is where you actually try.
## One key to rule the recovery
Consolidation concentrates risk on purpose one NAS, one repository format, one passphrase unlocking every
Consolidation concentrates risk on purpose - one NAS, one repository format, one passphrase unlocking every
cluster's history. That's a fine trade *if* the key management is treated with the same rigour as the data:
the passphrase lives offline, not in any repository it unlocks, and the recovery plan starts from "a bare
machine and the key", assuming nothing else survived. A backup encrypted with a key stored next to it is a
@@ -55,11 +55,11 @@ padlock with the key taped to the shackle.
## The principle
Trust receipts, not reports. Any system whose entire purpose is a future emergency backups, break-glass
accounts, failovers, restore paths is untested by definition on every normal day, so you have to
Trust receipts, not reports. Any system whose entire purpose is a future emergency - backups, break-glass
accounts, failovers, restore paths - is untested by definition on every normal day, so you have to
manufacture the test: drill it, on a schedule, with a real pass/fail, and treat a failed drill as a real
incident. The nightly green tick is the rumour. The drill is the receipt.
*Live across the homelab: per-cluster restic repositories on the NAS over SMB, nightly CronJobs, restore
drills run before each cutover thousands of files pulled back and verified per cluster, a stale-lock
drills run before each cutover - thousands of files pulled back and verified per cluster, a stale-lock
failure caught and self-healing added, the passphrase held offline.*
+14 -14
View File
@@ -1,7 +1,7 @@
---
title: "The first secret is the one you can't commit"
date: 2026-06-28
summary: "Secrets management has a bootstrap paradox: the credential that pulls every other secret can't itself live in git. Here's how a homelab fleet breaks the cycle zero plaintext secrets in any repo, and a clean rule for which is the one exception."
summary: "Secrets management has a bootstrap paradox: the credential that pulls every other secret can't itself live in git. Here's how a homelab fleet breaks the cycle - zero plaintext secrets in any repo, and a clean rule for which is the one exception."
tags: ["security", "gitops", "secrets", "kubernetes", "external-secrets"]
draft: false
hero: "/blog/secret-zero.webp"
@@ -13,12 +13,12 @@ Every "we do GitOps properly" story has a chicken-and-egg problem hiding in the
posts quietly skip it.
The pitch is clean: secrets never live in git. Instead, the cluster runs the [External Secrets
Operator](https://external-secrets.io) (ESO), which reads from a real secret store here, a self-hosted
[Infisical](https://infisical.com) and materialises a Kubernetes `Secret` for each app. Your repo only ever
Operator](https://external-secrets.io) (ESO), which reads from a real secret store - here, a self-hosted
[Infisical](https://infisical.com), and materialises a Kubernetes `Secret` for each app. Your repo only ever
contains an `ExternalSecret` manifest: a **pointer** ("give `immich` the value at key `IMMICH_DB_PASSWORD`"),
never a value. Beautiful. Auditable. Diff-able.
Except ESO has to authenticate to Infisical somehow. That's a credential. Where does *it* live?
Except - ESO has to authenticate to Infisical somehow. That's a credential. Where does *it* live?
## Secret zero
@@ -26,24 +26,24 @@ You can't store the bootstrap credential in git (that's the whole point), and yo
secret store (you need it *to reach* the secret store). This is **secret zero**: the one credential that the
entire chain hangs off, that has to be injected from outside the GitOps loop.
So you treat it as exactly that special, minimal, and out-of-band:
So you treat it as exactly that - special, minimal, and out-of-band:
- It's a **machine identity** scoped to *read-only*, and scoped per cluster. cave's ESO can't read alfred's
secrets and vice-versa. A leak is blast-radius-limited to one cluster's read path.
- It's applied **once, by hand** (or by a sealed bootstrap step), never committed. Everything downstream of it
is declarative.
- After it lands, a single `ClusterSecretStore` object which *is* in git, because it's just a pointer to the
store plus a reference to secret zero wires the whole cluster up.
- After it lands, a single `ClusterSecretStore` object, which *is* in git, because it's just a pointer to the
store plus a reference to secret zero - wires the whole cluster up.
From there the dam breaks in the good way: every app's `ExternalSecret` resolves through that store, ESO keeps
the `Secret` in sync, and your repo stays a map of *names*, not values.
<!-- DIAGRAM: secret-zero bootstrap chain out-of-band identity ClusterSecretStore ExternalSecrets app Secrets -->
<!-- DIAGRAM: secret-zero bootstrap chain - out-of-band identity -> ClusterSecretStore -> ExternalSecrets -> app Secrets -->
![Secret-zero bootstrap chain](/diagrams/secret-zero.svg?v=1)
## The discipline around the one exception
One out-of-band credential is fine. The trap is letting it sprawl or fumbling its rotation. Two rules earn
One out-of-band credential is fine. The trap is letting it sprawl, or fumbling its rotation. Two rules earn
their keep:
**Per-cluster, least-privilege identities.** It's tempting to mint one powerful identity and reuse it
@@ -53,22 +53,22 @@ every node.
**Never revoke a shared identity until every consumer has moved off it.** This sounds obvious and is the
single easiest way to take the fleet down. When you split a shared identity into per-cluster ones, the old one
stays valid until you've *verified* each cluster is happily authenticating on its own then, and only then,
stays valid until you've *verified* each cluster is happily authenticating on its own - then, and only then,
you revoke. Revoke-first-verify-later turns a tidy-up into an outage.
## What still belongs in git (encrypted)
A purist would stop here, but reality has a few things that genuinely need to live *in* the repo a value a
A purist would stop here, but reality has a few things that genuinely need to live *in* the repo: a value a
bootstrap step reads before ESO is even running. For those, the answer isn't "commit it in plaintext and feel
bad," it's **SOPS + age**: the value is encrypted in git, decryptable only by a key that lives on the
operators' machines (and the cluster), never in the repo. Same principle as secret zero the *decryption* key
is the out-of-band thing applied to the handful of values that can't wait for the operator to spin up.
operators' machines (and the cluster), never in the repo. Same principle as secret zero (the *decryption* key
is the out-of-band thing) applied to the handful of values that can't wait for the operator to spin up.
## The shape of it
The win isn't any one tool. It's the shape: **exactly one** credential lives outside GitOps, it's read-only
and per-cluster, and it's the seed the whole tree grows from. Everything else is a pointer you can show
your worst enemy. When someone asks "where are your secrets?", the honest answer is "in the store the repo
your worst enemy. When someone asks "where are your secrets?", the honest answer is "in the store - the repo
just knows their *names*," and that one sentence is the whole security model.
*This runs across a five-cluster Talos homelab; the ESO + Infisical wiring, the per-cluster read-only
+7 -7
View File
@@ -1,7 +1,7 @@
---
title: "Shipping this site: GitOps from a homelab to the public internet"
date: 2026-06-15
summary: "How this portfolio is built and served Astro to a container image, a self-hosted Gitea registry, ArgoCD, and a Cloudflare Tunnel with security as acceptance criteria, not polish."
summary: "How this portfolio is built and served - Astro to a container image, a self-hosted Gitea registry, ArgoCD, and a Cloudflare Tunnel - with security as acceptance criteria, not polish."
tags: ["gitops", "astro", "homelab", "security"]
hero: "/blog/shipping-this-site.webp"
heroAlt: "A glowing data container travels a luminous rail from a small server rack toward a softly glowing globe of Earth, wrapped by a reconcile loop."
@@ -10,16 +10,16 @@ heroAlt: "A glowing data container travels a luminous rail from a small server r
This site is a static Astro build, and that's the least interesting thing about it. What matters
is how it reaches you: served from my homelab Kubernetes cluster over a Cloudflare Tunnel, shipped
the way I'd ship anything I actually cared about an immutable image, pinned by digest, reconciled
the way I'd ship anything I actually cared about - an immutable image, pinned by digest, reconciled
by GitOps. No special case for "it's just a website."
## The pipeline
The build is baked into a hardened `nginx-unprivileged` image and pushed to a **self-hosted public
Gitea registry** deliberately a separate instance from the private one holding my infrastructure
Gitea registry**, deliberately a separate instance from the private one holding my infrastructure
code, so the public artifact and the private estate never share a trust boundary. The image digest
is then pinned in a private `home-ops` repo, **ArgoCD** reconciles that repo onto the cluster, and
a **Cloudflare Tunnel** exposes exactly one service this site outbound-only.
a **Cloudflare Tunnel** exposes exactly one service, this site, outbound-only.
Follow the chain and notice what's missing. No open ports: the tunnel dials out. No server runtime:
the output is static files behind nginx. No registry credential on the cluster: the public package
@@ -32,9 +32,9 @@ The discipline that made it work was treating security as a checklist to *pass*,
gesture at. The site didn't ship until every box was ticked:
```text
[x] Static output no server runtime to attack
[x] Static output - no server runtime to attack
[x] Strict CSP, no unsafe-inline / unsafe-eval
[x] Self-hosted fonts zero third-party requests
[x] Self-hosted fonts - zero third-party requests
[x] No secrets in the client bundle (verified by build-time grep)
[x] Outbound-only tunnel, single hostname, no catch-all
```
@@ -46,7 +46,7 @@ verify, and a failing item blocks the ship. "Pretty secure" isn't a state you ca
## The principle
The site *is* the argument. A platform engineer's portfolio should demonstrate the discipline it
advertises, and "it's a static page" is no excuse to skip the rigour it's the cheapest possible
advertises, and "it's a static page" is no excuse to skip the rigour - it's the cheapest possible
place to practise it. If the pipeline behind a brochure site is immutable, verified, and
zero-trust, that's not overkill. That's the standard, rehearsed where the stakes are low so it
holds where they aren't.
+1 -1
View File
@@ -1,7 +1,7 @@
---
title: "Six days of nothing, and nothing noticed"
date: 2026-08-17
summary: "A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied - because nothing was watching the thing that had actually failed."
summary: "A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed."
tags: ["observability", "monitoring", "reliability", "alerting", "homelab"]
draft: false
hero: "/blog/six-days-of-silence.webp"
+24 -27
View File
@@ -1,31 +1,29 @@
---
title: "Powered off is a state, not an incident"
date: 2026-08-02
summary: "Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear."
summary: "Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night, and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear."
tags: ["homelab", "operations", "talos", "sustainability", "architecture"]
draft: false
hero: "/blog/sleep-half-the-fleet.webp"
heroAlt: "A dark tower of circuit-traced architecture at night most windows unlit and resting, a few glowing where the core still works"
heroAlt: "A dark tower of circuit-traced architecture at night - most windows unlit and resting, a few glowing where the core still works"
---
Homelabs run 24/7 out of habit. The habit comes with a scoreboard `uptime` counters climbing into the
hundreds of days, worn as proof of seriousness and with two feelings nobody says out loud: the guilt of
Homelabs run 24/7 out of habit. The habit comes with a scoreboard - `uptime` counters climbing into the
hundreds of days, worn as proof of seriousness, and with two feelings nobody says out loud: the guilt of
turning a server off, as if a dark machine means you weren't serious; and the quieter fear that if you do
turn it off, it won't come back. Somewhere on that box, the fear whispers, is a state nobody wrote down,
held together by warmth.
Meanwhile the actual work is bursty. GPU inference runs when a request arrives. Transcode runs in batches
and finishes. The hardware idles around the clock for workloads that occupy it a fraction of the time
real watts, real fan-hours, real heat, spent keeping silicon warm for nobody.
and finishes. The hardware idles around the clock for workloads that occupy it a fraction of the time - real watts, real fan-hours, real heat, spent keeping silicon warm for nobody.
Last night I shut down half the fleet on purpose. Three machines two of them carrying an RTX 5080 each
were upgraded in the afternoon, verified healthy in the evening, and powered off before midnight. Nothing
Last night I shut down half the fleet on purpose. Three machines - two of them carrying an RTX 5080 each - were upgraded in the afternoon, verified healthy in the evening, and powered off before midnight. Nothing
broke, nothing was lost, and the fleet's books record the event as a state, not a failure. Think of a
building at night: the porter walks the floors, the lights go off in the wings nobody's using, and the
building isn't failing. It's resting and the porter writes it in the log.
building isn't failing. It's resting, and the porter writes it in the log.
<!-- DIAGRAM: three node power states in a row READY (lit) OFF-BY-CHOICE (dark, a single power-button arrow back to Ready) and, set apart, PARKED-BROKEN (dark, a screwdriver arrow as its only exit); beneath all three states one continuous, unbroken document layer (git: machine config · CNI inline · workloads · credentials) the document layer is what's highlighted, the power states sit on top of it as incidental -->
![One document persists beneath every power state Ready, off by choice, parked broken](/diagrams/sleep-half-the-fleet.svg)
<!-- DIAGRAM: three node power states in a row - READY (lit) <-> OFF-BY-CHOICE (dark, a single power-button arrow back to Ready) and, set apart, PARKED-BROKEN (dark, a screwdriver arrow as its only exit); beneath all three states one continuous, unbroken document layer (git: machine config / CNI inline / workloads / credentials) - the document layer is what's highlighted, the power states sit on top of it as incidental -->
![One document persists beneath every power state - Ready, off by choice, parked broken](/diagrams/sleep-half-the-fleet.svg)
## Make the node a document, and power becomes a detail
@@ -34,18 +32,18 @@ post](/blog/no-ssh): each of these machines is a single-node Talos cluster whose
declared. The machine config pins the disks by serial and the network interfaces by MAC; the CNI ships
inside that same document as an inline manifest; the workloads arrive from git the moment the node reports
Ready, reconciled by the fleet's GitOps controller; the credentials are substituted in at render time from
an encrypted file. There is no state that exists only on the box. The node *is* a document power is just
an encrypted file. There is no state that exists only on the box. The node *is* a document - power is just
whether it's currently being read.
That property is what made this week's sequence unremarkable. Three nodes had their network datapath
swapped kube-proxy out, Cilium's eBPF replacement in and their registry credentials rotated, one reboot
swapped - kube-proxy out, Cilium's eBPF replacement in, and their registry credentials rotated, one reboot
each. Every check came back green: GitOps fully synced, secrets flowing, every endpoint answering. And
then, the same night, all three were shut down. Upgraded, then slept. Nothing about the upgrade is lost to
the power state: it's on disk, in the config, in git. When the power button is next pressed, each node
boots into exactly the machine its documents describe including everything that changed yesterday.
boots into exactly the machine its documents describe - including everything that changed yesterday.
How they went down matters as much as the fact that they could. `talosctl shutdown` is a clean, API-driven
stop: workloads terminated properly, etcd closed cleanly, the platform powering itself off confirmed dark
stop: workloads terminated properly, etcd closed cleanly, the platform powering itself off - confirmed dark
before anyone walked away. A graceful shutdown is what makes the next boot boring. Pull the plug instead
and you've converted a healthy state into a small forensic exercise for future-you.
@@ -53,16 +51,16 @@ and you've converted a healthy state into a small forensic exercise for future-y
Four of the six machines are dark tonight, and the fleet's records refuse to treat them as one category.
Three robin, kate and wgirl are **off by choice**: healthy at shutdown, upgraded that same day, one
Three (robin, kate and wgirl) are **off by choice**: healthy at shutdown, upgraded that same day, one
press of a power button from Ready. The fourth, elfastc, is **parked broken**: dark because of a hardware
fault, waiting on a screwdriver, not a switch. From the network's point of view the four are identical no
fault, waiting on a screwdriver, not a switch. From the network's point of view the four are identical: no
link light, no metrics, nothing to distinguish rest from wreckage. From the operator's they are different
states with different exits, and the roster records which is which, in the same versioned pages as
everything else.
This is the porter's log. A dark wing and a broken lift look the same from the street; the difference is
written down at the desk. Six months from now, "why is this node off?" must have an answer that isn't
archaeology because a fleet that can't tell *resting* from *broken* will eventually treat one as the
archaeology, because a fleet that can't tell *resting* from *broken* will eventually treat one as the
other, and both mistakes are expensive: rebuilding a healthy node because nobody trusted it, or trusting a
broken one because it merely looked asleep.
@@ -72,27 +70,26 @@ Sleep isn't free, and pretending otherwise is how the 24/7 habit wins the argume
both are stated in the books.
First: what lives on a sleeping node is dormant, not migrated. The fleet's LLM backend lives on wgirl's
GPU, and it sleeps when she does. The assistant app that depends on it stays up it runs on a node that's
awake but until power-on it has no model to call. That is a genuine loss of capability, accepted
GPU, and it sleeps when she does. The assistant app that depends on it stays up - it runs on a node that's
awake, but until power-on it has no model to call. That is a genuine loss of capability, accepted
deliberately, recorded next to the state that caused it. If the trade ever stops being worth it, the fix is
a power button, not a redesign.
Second: monitoring fires as a node goes silent. The metrics stream stops mid-sentence and the alert raises,
exactly as it should the monitoring has no way to know the silence was chosen, and it must never assume
exactly as it should - the monitoring has no way to know the silence was chosen, and it must never assume
so. The wrong response is to call that a false alarm. It isn't. It's the system being honest about a
deliberate act, and the deliberate act is written down where the alert can be checked against it. The
porter doesn't disable the alarm panel for the dark wings he checks the panel against the log.
porter doesn't disable the alarm panel for the dark wings - he checks the panel against the log.
## The principle
Design for off. A node you can power down without ceremony is a node whose whole truth lives outside it
in documents, in git, in the log — and that is precisely the property you need on the day a node goes down
Design for off. A node you can power down without ceremony is a node whose whole truth lives outside it - in documents, in git, in the log, and that is precisely the property you need on the day a node goes down
without asking. The fleet you can put to sleep on purpose and the fleet that shrugs off losing a node in
anger are the same fleet; the two abilities are one property. Uptime measures how long a machine has been
running. It says nothing about how well you hold it. A building isn't failing when its lights go out at
night it's failing when nobody dares touch the switch.
night - it's failing when nobody dares touch the switch.
*Live in the fleet's books tonight: three of six Talos nodes robin and wgirl (an RTX 5080 each) and kate
(the AI sandbox) upgraded to the new datapath, verified green, then gracefully shut down through the API
*Live in the fleet's books tonight: three of six Talos nodes - robin and wgirl (an RTX 5080 each) and kate
(the AI sandbox) - upgraded to the new datapath, verified green, then gracefully shut down through the API
and recorded off-by-choice; elfastc parked-broken awaiting a screwdriver; cave and alfred carrying the
always-on core.*
+19 -21
View File
@@ -1,7 +1,7 @@
---
title: "SNAT ate my source IP"
date: 2026-07-14
summary: "A LoadBalancer service with the default traffic policy rewrites every incoming packet's source to the node's own address so by the time a network policy sees it, the real client is gone. You cannot allowlist a sender the network has already erased."
summary: "A LoadBalancer service with the default traffic policy rewrites every incoming packet's source to the node's own address, so by the time a network policy sees it, the real client is gone. You cannot allowlist a sender the network has already erased."
tags: ["security", "networking", "cilium", "kubernetes", "load-balancing"]
draft: false
hero: "/blog/snat-ate-my-source.webp"
@@ -13,17 +13,17 @@ Here's the uncomfortable default: give a Kubernetes service a LoadBalancer IP, l
address rewritten to the node's own**. Not by an attacker. By the load balancer, on purpose, as
routine plumbing.
I found out the way you always find out by writing a network policy for it. LAN clients live on
I found out the way you always find out - by writing a network policy for it. LAN clients live on
`10.0.11.0/24`, so I wrote a `fromCIDR 10.0.11.0/24` allow. Applied cleanly, showed `Valid=True`,
dropped every real user. Hubble told me why: the traffic wasn't arriving from the LAN at all. Cilium
tagged the source `world` the identity of *anywhere*, the one you can't restrict without
tagged the source `world` - the identity of *anywhere*, the one you can't restrict without
restricting everything.
It's a letter that reaches you carrying the sorting office's return address instead of the sender's.
The sorting office isn't lying to you; it's how the machinery forwards mail. But you can't write a
"letters from Alice only" rule when every envelope on your doormat says it came from the depot.
<!-- DIAGRAM: client LB VIP SNAT at the node (source rewritten) policy sees `world`, vs ETP:Local preserving the client address -->
<!-- DIAGRAM: client -> LB VIP -> SNAT at the node (source rewritten) -> policy sees `world`, vs ETP:Local preserving the client address -->
![SNAT rewriting the client source before policy evaluation, and ETP:Local preserving it](/diagrams/snat-ate-my-source.svg)
## Why the depot stamps over the sender
@@ -35,63 +35,61 @@ destroying the source.
The consequence lands exactly where you can't see it coming: **policies bind to what arrives, not to
what was sent.** The client's packet left home with an honest return address; the depot stamped over
it in transit; your policy evaluated after the rewrite never meets the client at all. There is no
it in transit; your policy - evaluated after the rewrite - never meets the client at all. There is no
rule you can write for an identity the network erased one hop earlier.
## Preserve the sender, then commit it
The fix is one line: `externalTrafficPolicy: Local`. Only nodes running a backend pod accept
the traffic, no cross-node forwarding happens, no SNAT is needed the client's real address survives
the traffic, no cross-node forwarding happens, no SNAT is needed, and the client's real address survives
to policy evaluation. Now `fromCIDR 10.0.11.0/24` matches LAN clients, and a cross-cluster peer shows
up as its node IP the peer *node*, not the peer pod, because separate clusters masquerade pod
traffic on the way out pin-downable with a `/32`. That's how the LLM gateway's ingress rule got to
up as its node IP (the peer *node*, not the peer pod, because separate clusters masquerade pod
traffic on the way out), pin-downable with a `/32`. That's how the LLM gateway's ingress rule got to
be a single line: one legitimate consumer, the agent cluster next door, exactly one `/32` allowed in.
Here's the one that costs a 2am. I made that change with `kubectl patch` quick, out-of-band, worked
Here's the one that costs a 2am. I made that change with `kubectl patch` - quick, out-of-band, worked
immediately. The service was GitOps-managed with self-heal on. Within seconds ArgoCD noticed the live
object differed from git and put it back the way the repo said, which re-enabled the SNAT, which
re-broke the policy, which took DNS down with it. **The change must be committed**, or the platform
will politely undo your fix while you sleep.
One caveat that keeps it honest: some paths still rewrite. A client arriving through a
tailnet subnet router gets masqueraded at the routing node even with `Local` set it lands as
tailnet subnet router gets masqueraded at the routing node even with `Local` set - it lands as
`world` regardless. Some envelopes pass through a second depot you don't control.
## When the depot closes, every envelope changes
This week the fleet finished swapping kube-proxy for Cilium's eBPF replacement. Same services, same
addresses, same policies and a traefik that had answered LAN probes for months went dark, under a
addresses, same policies, and a traefik that had answered LAN probes for months went dark, under a
policy nobody touched.
It was never supposed to answer. That traefik fronts zero routes, and the tightest floor guards it
a policy admitting `host`, because kubelet probes need it, and nothing else. But kube-proxy's SNAT
had been dressing LAN probes as `host` — a different depot, a different stamp, the same disease. The
It was never supposed to answer. That traefik fronts zero routes, and the tightest floor guards it: a policy admitting `host`, because kubelet probes need it, and nothing else. But kube-proxy's SNAT
had been dressing LAN probes as `host`: a different depot, a different stamp, the same disease. The
replacement closed the depot. Every envelope arrived carrying its real return address, the probes
landed as `world`, and the policy dropped them exactly as written. Nothing broke. The policy
landed as `world`, and the policy dropped them, exactly as written. Nothing broke. The policy
finally saw the truth, and the truth matched the original intent: answer no one.
But if you didn't know *why* it used to answer, this is the morning you file an outage ticket and
"fix" a policy that was never wrong. The rewriting machinery is itself a moving part. Change the
plumbing and every identity changes with it which is why only observed flows stay true, and
plumbing and every identity changes with it, which is why only observed flows stay true, and
remembered ones quietly expire.
## Read the envelope before you write the rule
The real lesson isn't the one-line fix. It's that I wrote a policy for the traffic I *imagined*
clients on the LAN, arriving as themselves — instead of the traffic that *arrived*. The packet's
The real lesson isn't the one-line fix. It's that I wrote a policy for the traffic I *imagined* - clients on the LAN, arriving as themselves - instead of the traffic that *arrived*. The packet's
story gets rewritten at every hop: SNAT here, DNAT there, a masquerade at a routing boundary. The
sender's truth and the receiver's truth are different documents.
So the discipline, before any allow rule exists: **observe first.** Open Hubble, watch the real
flows, and note the identity the traffic actually carries when it reaches the endpoint `world`,
flows, and note the identity the traffic actually carries when it reaches the endpoint - `world`,
`host`, `cluster`, a pod label, a CIDR. Then write the rule for *that*. A policy written from the
architecture diagram is a guess; a policy written from observed flows is a fact.
## The principle
A network policy is a doorman checking return addresses, and the postal system rewrites return
addresses as a matter of course. You don't secure what was sent nobody ever sees what was sent.
addresses as a matter of course. You don't secure what was sent - nobody ever sees what was sent.
You secure what arrives. Go and look at the envelope first.
*Bitten and fixed on a six-cluster Talos fleet running Cilium 1.19 kube-proxy replaced by the eBPF
*Bitten and fixed on a six-cluster Talos fleet running Cilium 1.19 - kube-proxy replaced by the eBPF
datapath fleet-wide, MetalLB retired, and a standing rule: Hubble before policy, every time.*
+14 -14
View File
@@ -1,50 +1,50 @@
---
title: "Stage every lock before you turn one"
date: 2026-08-02
summary: "Turning off anonymous pulls on the registry a fleet boots from is a change that only fails at the next boot and a node that can't pull at boot may not come back to tell you. So the flip is split in time: the credential staged inert in every node's config, each node rebooted and proven with a real pull, and the door closed only when the last key has turned."
summary: "Turning off anonymous pulls on the registry a fleet boots from is a change that only fails at the next boot, and a node that can't pull at boot may not come back to tell you. So the flip is split in time: the credential staged inert in every node's config, each node rebooted and proven with a real pull, and the door closed only when the last key has turned."
tags: ["security", "registry", "kubernetes", "talos", "homelab"]
draft: false
hero: "/blog/stage-the-locks.webp"
heroAlt: "A corridor of doors propped open, new lock cylinders and freshly cut keys glowing on a locksmith's bench every lock staged, none yet turned"
heroAlt: "A corridor of doors propped open, new lock cylinders and freshly cut keys glowing on a locksmith's bench - every lock staged, none yet turned"
---
Every node in this fleet pulls its images through one registry: a pull-through mirror fronting five upstreams behind a single host. Anonymous read has been on since the day it went up the bootstrap-era convenience you stop noticing, right up until you write the words *production posture* and realise the registry the entire estate boots from will hand an image to anyone who asks. The fix is not exotic: a named pull identity for the fleet, anonymous off.
Every node in this fleet pulls its images through one registry: a pull-through mirror fronting five upstreams behind a single host. Anonymous read has been on since the day it went up - the bootstrap-era convenience you stop noticing, right up until you write the words *production posture* and realise the registry the entire estate boots from will hand an image to anyone who asks. The fix is not exotic: a named pull identity for the fleet, anonymous off.
And the reflex for shipping a fix like that is the reflex that works for everything else. Make the change, watch it, roll it back if it misbehaves. Flip the policy, delete a pod, watch the pod re-pull. Green means done.
Except this change doesn't fail when you make it. It fails at the *next boot* and a node that can't pull at boot may not come back to tell you about it. No test run after the flip proves the fleet safe, because the only test that counts is a reboot, and the reboots that count are the ones you didn't plan. So the change gets split in time, the way a locksmith re-keys an occupied building: fit every cylinder and cut every key while the doors are propped open, try each key in its own door, and only when the last key turns do the props come out.
Except this change doesn't fail when you make it. It fails at the *next boot*, and a node that can't pull at boot may not come back to tell you about it. No test run after the flip proves the fleet safe, because the only test that counts is a reboot, and the reboots that count are the ones you didn't plan. So the change gets split in time, the way a locksmith re-keys an occupied building: fit every cylinder and cut every key while the doors are propped open, try each key in its own door, and only when the last key turns do the props come out.
<!-- DIAGRAM: a three-stage timeline, left to right. STAGE: one pull credential fanned out inert into six node-config repos (git icon), the registry door propped open (anonymous read still ON). PROVE: per-node reboot onto the auth, then a gated pull of a mirror-only image through the node's own runtime three of six ticked, one node parked for hardware repair. FLIP: anonymous read switched off at the registry ONLY at six of six props out, door latches. Highlight the per-node gate, not the flip. -->
<!-- DIAGRAM: a three-stage timeline, left to right. STAGE: one pull credential fanned out inert into six node-config repos (git icon), the registry door propped open (anonymous read still ON). PROVE: per-node reboot onto the auth, then a gated pull of a mirror-only image through the node's own runtime - three of six ticked, one node parked for hardware repair. FLIP: anonymous read switched off at the registry ONLY at six of six - props out, door latches. Highlight the per-node gate, not the flip. -->
![Stage the credential inert in every node's config, prove each node with a real pull, flip anonymous off only on full coverage](/diagrams/stage-the-locks.svg)
## The fallback you'd bet on doesn't hold
On paper the naive flip looks survivable. Every node's registry config lists the mirror first and the true upstream second, so if the mirror answered a 401 you'd expect the runtime to shrug and pull direct. That expectation is exactly what the red-team refused to stake a fleet on: containerd's fallback behaviour on an auth failure is unreliable in practice known, numbered upstream issues (#7321, #9997), not a hypothesis. And the trap has good manners. The flip goes green on the day, because every running pod already holds its images. The failure waits, patient, for the next power blip then turns it into an outage.
On paper the naive flip looks survivable. Every node's registry config lists the mirror first and the true upstream second, so if the mirror answered a 401 you'd expect the runtime to shrug and pull direct. That expectation is exactly what the red-team refused to stake a fleet on: containerd's fallback behaviour on an auth failure is unreliable in practice - known, numbered upstream issues (#7321, #9997), not a hypothesis. And the trap has good manners. The flip goes green on the day, because every running pod already holds its images. The failure waits, patient, for the next power blip - then turns it into an outage.
So the red-team moved the flip to a window where every reboot is free: a fleet power-on, nodes coming up anyway, someone watching, a stumbling node getting attention instead of becoming a mystery at two in the morning. That's the first half of the split this class of change never gets to surface its failure on a reboot you didn't choose.
So the red-team moved the flip to a window where every reboot is free: a fleet power-on, nodes coming up anyway, someone watching, a stumbling node getting attention instead of becoming a mystery at two in the morning. That's the first half of the split - this class of change never gets to surface its failure on a reboot you didn't choose.
## Cut every key while the doors are propped
The second half is staging, and staging happens well before enforcement. The credential exists now. It lives in an encrypted environment file beside each cluster's config and is substituted into the node's machine configuration when the config renders the committed file carries a placeholder, never the value. That auth block went into all six node repos in one reviewed pass, while anonymous read stayed on. Inert. A credential the registry never challenges you for does nothing at all; it sits in git, costing nothing, waiting.
The second half is staging, and staging happens well before enforcement. The credential exists now. It lives in an encrypted environment file beside each cluster's config and is substituted into the node's machine configuration when the config renders - the committed file carries a placeholder, never the value. That auth block went into all six node repos in one reviewed pass, while anonymous read stayed on. Inert. A credential the registry never challenges you for does nothing at all; it sits in git, costing nothing, waiting.
Two costs are worth naming while it waits. Committing is free but *loading* isn't: on this platform, mirror endpoints reload live while registry auth only takes effect at a reboot which is precisely why the flip has to ride reboots that were already happening. And the rotation bill went into the decision record at decision time: this is one shared pull identity across six nodes, so rotating it later means six config edits and a fleet of reboots. Accepted, consciously, in writing not discovered mid-incident two years from now.
Two costs are worth naming while it waits. Committing is free but *loading* isn't: on this platform, mirror endpoints reload live while registry auth only takes effect at a reboot, which is precisely why the flip has to ride reboots that were already happening. And the rotation bill went into the decision record at decision time: this is one shared pull identity across six nodes, so rotating it later means six config edits and a fleet of reboots. Accepted, consciously, in writing, not discovered mid-incident two years from now.
## Try each key in its own door
Then each node proves itself, one at a time. The gate is not "it booted". Booting proves nothing the doors are still propped, so anonymous pulls succeed regardless, and cached images would mask a broken credential anyway. The gate is a real pull, driven through the node's own container runtime via the management API, for an image that exists only on the mirror and nowhere upstream. A pass means one thing: this node presented the new credential and the registry honoured it. No fallback can fake that result, and no cache can serve it.
Then each node proves itself, one at a time. The gate is not "it booted". Booting proves nothing - the doors are still propped, so anonymous pulls succeed regardless, and cached images would mask a broken credential anyway. The gate is a real pull, driven through the node's own container runtime via the management API, for an image that exists only on the mirror and nowhere upstream. A pass means one thing: this node presented the new credential and the registry honoured it. No fallback can fake that result, and no cache can serve it.
That's the discipline in a sentence: readiness is never inferred from the node coming back. Readiness is the key turning in the door, observed, per node.
## The props stay in until the last key turns
Where it stands, honestly: three of the six nodes are staged and gated as of this week, all in a single window and cheaply, because each node's credential reboot rode a reboot that another migration was paying for anyway. Changes that share a reboot cost less than changes that each demand their own; batching them into one window is among the cheapest wins fleet operations offers.
Where it stands, honestly: three of the six nodes are staged and gated as of this week, all in a single window, and cheaply, because each node's credential reboot rode a reboot that another migration was paying for anyway. Changes that share a reboot cost less than changes that each demand their own; batching them into one window is among the cheapest wins fleet operations offers.
The flip itself is still blocked. Deliberately. One node is parked awaiting a hardware repair, so the door stays propped for everyone until that node passes its gate too. That isn't the plan slipping it *is* the plan. The asymmetry does all the work: a staged lock you haven't turned costs nothing, while a turned lock with one unproven key costs a boot loop on the worst morning of the quarter. With arithmetic that lopsided, waiting isn't caution. It's reading the numbers.
The flip itself is still blocked. Deliberately. One node is parked awaiting a hardware repair, so the door stays propped for everyone until that node passes its gate too. That isn't the plan slipping - it *is* the plan. The asymmetry does all the work: a staged lock you haven't turned costs nothing, while a turned lock with one unproven key costs a boot loop on the worst morning of the quarter. With arithmetic that lopsided, waiting isn't caution. It's reading the numbers.
## The principle
Registry auth is one instance of a wider class: changes whose failure surfaces only at the next boot, restart, or failover where the machine that would report the failure is the machine the failure takes down. For that class, make-it-and-watch is not a strategy, because there is nothing to watch until rollback is no longer on the table. The split is the strategy. Stage the change inert, in config, where it can be reviewed and reverted for free. Prove it per node, under conditions where failure is cheap and attended. Enforce only on full coverage never on most of it.
Registry auth is one instance of a wider class: changes whose failure surfaces only at the next boot, restart, or failover - where the machine that would report the failure is the machine the failure takes down. For that class, make-it-and-watch is not a strategy, because there is nothing to watch until rollback is no longer on the table. The split is the strategy. Stage the change inert, in config, where it can be reviewed and reverted for free. Prove it per node, under conditions where failure is cheap and attended. Enforce only on full coverage - never on most of it.
Stage the locks first; turn them last.
*Live on a six-node Talos fleet: a zot pull-through mirror fronting five upstreams at one host, the fleet pull credential staged by sops-encrypted substitution into all six cluster repos, three nodes rebooted and gated with a mirror-only image pull and anonymous read still on, deliberately, until the sixth key turns.*
*Live on a six-node Talos fleet: a zot pull-through mirror fronting five upstreams at one host, the fleet pull credential staged by sops-encrypted substitution into all six cluster repos, three nodes rebooted and gated with a mirror-only image pull, and anonymous read still on, deliberately, until the sixth key turns.*
+10 -11
View File
@@ -1,7 +1,7 @@
---
title: "Every pod holds a key to a door it never opens"
date: 2026-06-23
summary: "Least privilege for Kubernetes workloads doesn't start with an RBAC role it starts with revoking the API token every pod silently carries, then layering identity, non-root, and Pod Security on top."
summary: "Least privilege for Kubernetes workloads doesn't start with an RBAC role - it starts with revoking the API token every pod silently carries, then layering identity, non-root, and Pod Security on top."
tags: ["kubernetes", "security", "least-privilege", "rbac", "service-accounts"]
draft: false
hero: "/blog/workload-least-privilege.webp"
@@ -10,23 +10,23 @@ heroAlt: "Rows of glowing cube-pods each clutch a keycard, facing a sealed vault
---
Open a shell in almost any Kubernetes pod and look in `/var/run/secrets/kubernetes.io`. There's a token
there a live credential for the cluster API, mounted automatically, signed and ready. Now ask the harder
there - a live credential for the cluster API, mounted automatically, signed and ready. Now ask the harder
question: does this workload ever actually *call* the Kubernetes API? For a media server, a credential vault,
a DNS sinkhole, a search proxy the answer is no. Never. Not once.
a DNS sinkhole, a search proxy - the answer is no. Never. Not once.
So every one of those pods is holding a key to a door it never opens. That's not least privilege. That's a
standing credential waiting to be stolen.
![Workload least privilege each pod gets its own ServiceAccount with no API token, runs non-root with capabilities dropped, inside a namespace whose Pod Security level is pinned; the cluster API it never calls stays out of reach](/diagrams/workload-least-privilege.svg?v=2)
![Workload least privilege - each pod gets its own ServiceAccount with no API token, runs non-root with capabilities dropped, inside a namespace whose Pod Security level is pinned; the cluster API it never calls stays out of reach](/diagrams/workload-least-privilege.svg?v=2)
## Identity first, RBAC second
The reflex when someone says "least-privilege RBAC" is to start writing Roles. But a Role narrows what a
credential *can do* it does nothing about a credential that shouldn't exist. The first move is cheaper and
credential *can do* - it does nothing about a credential that shouldn't exist. The first move is cheaper and
safer: give every workload its **own** ServiceAccount instead of the shared namespace default, and set
`automountServiceAccountToken: false`.
That single line changes no runtime behaviour no capabilities, no user, no restart semantics because
That single line changes no runtime behaviour - no capabilities, no user, no restart semantics - because
the app wasn't using the token anyway. You've simply stopped handing out a key. The handful of components
that genuinely talk to the API (the GitOps controller, the load-balancer, the secrets operator) get a
scoped Role and keep their token. Everything else gets an identity and nothing else. It's the highest
@@ -39,15 +39,15 @@ full Linux capability set is one escape away from the host, regardless of how fe
So the next layer is the container itself: run as a non-root UID, drop **all** capabilities and add back only
the few a workload truly needs, forbid privilege escalation, and pin a `RuntimeDefault` seccomp profile.
This is where honesty matters. Some images anything running several processes under an init system, or the
classic "start as root, drop to a user" pattern break the moment you drop capabilities, because something
This is where honesty matters. Some images - anything running several processes under an init system, or the
classic "start as root, drop to a user" pattern - break the moment you drop capabilities, because something
inside still expects to `chown` a directory or bind a port. The lesson isn't "give up"; it's that non-root is
a *per-app, verified* change, while own-identity-and-no-token is a *universal, safe* one. Do the free thing
everywhere; do the careful thing where you've tested it.
## Make the floor non-negotiable
Authoring all of this per workload is good, but authoring isn't enforcing a future manifest can forget. The
Authoring all of this per workload is good, but authoring isn't enforcing - a future manifest can forget. The
enforcement floor is **Pod Security Admission**, set at the namespace. Pin each namespace's level explicitly
rather than inheriting a cluster default you can't see: `baseline` where root init-containers still live,
`restricted` for the apps that have earned it, and a documented `privileged` exception for the genuine cases
@@ -56,8 +56,7 @@ restricted profile, and every apply tells you exactly which workload is one refa
## The principle
Least privilege isn't a pile of Roles. It's a direction: start by removing the access that nothing is using
the auto-mounted token first — then narrow what genuinely remains, then make the boundary enforce itself. Give
Least privilege isn't a pile of Roles. It's a direction: start by removing the access that nothing is using - the auto-mounted token first - then narrow what genuinely remains, then make the boundary enforce itself. Give
every workload its own identity, hand it no credential it doesn't need, strip it to the smallest process it can
be, and let the namespace refuse anything bigger. The credential you never mount can't leak, can't be replayed,
and can't be the thing you forgot to revoke.
+1 -1
View File
@@ -14,6 +14,6 @@ export const fleetFacts = [
// Node roster for the constellation (positions live in the SVG component).
export const fleetNodes = {
hub: { name: "cave", caption: "139 apps · reconciling" },
hub: { name: "cave" },
satellites: ["alfred", "robin", "kate", "wgirl", "elfastc"],
} as const;