blog: publish three posts — backup restore drills, the bastion pattern, split-horizon DNS-01 (heroes + diagrams)
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: "The bastion is a pattern, not a box"
|
||||
date: 2026-07-03
|
||||
summary: "Admin credentials accumulate on whatever machine you happen to work from, until your daily driver is the real control plane of the fleet. A bastion fixes that — not by being special hardware, but by being the one place the keys live and the discipline that they never leave it."
|
||||
tags: ["security", "operations", "homelab", "tailscale", "access"]
|
||||
hero: "/blog/bastion-is-a-pattern.webp?v=1"
|
||||
heroAlt: "A lone illuminated harbour pilot station on a dark sea, thin beams of cyan and magenta light reaching out to distant vessels — the charts and keys stay at the station, the ships only transit past it."
|
||||
draft: false
|
||||
---
|
||||
|
||||
|
||||
Here's the uncomfortable default: your fleet's admin credentials live wherever you last needed them. A
|
||||
kubeconfig on the laptop, another copy on the desktop, an age key pasted onto a third machine "just for one
|
||||
decrypt", SSH keys sprouting wherever a clone was convenient. Nobody decided this. It accumulates.
|
||||
|
||||
Which means your **daily driver is the real control plane** — the machine that browses the web, opens email
|
||||
attachments and runs whatever you installed last week is also the machine that can wipe a cluster. Lose it,
|
||||
rebuild it, or compromise it, and you've done all three to the fleet.
|
||||
|
||||
The inversion is old and it still works: pick *one* point, put every key there, and make the rule that they
|
||||
never leave it. Not a hardened appliance. A pattern.
|
||||
|
||||
<!-- DIAGRAM: one bastion node holding keys/contexts/repos, lines out to the clusters + git; laptops connect only THROUGH it -->
|
||||
|
||||
|
||||
## Everything transits, nothing homes anywhere else
|
||||
|
||||
Think of a harbour pilot station. Every ship entering the harbour is guided by a pilot, but the charts and the
|
||||
local knowledge never board the ships — they stay at the station, and the vessels transit past it. That's the
|
||||
whole design: the dangerous knowledge has exactly one home, and traffic comes *to* it.
|
||||
|
||||
Mine is nothing special on purpose: a small Debian VM — four cores, no GPU passthrough, because a bastion needs
|
||||
none. What makes it the bastion is what it holds. One kubeconfig with a context for every cluster. The
|
||||
talosconfig for the OS layer beneath them. The SOPS age key that decrypts the fleet's secrets, sitting in
|
||||
exactly one place at `~/.config/sops/age/keys.txt`. A checkout of every ops repo. And a toolchain pinned
|
||||
deliberately — `talosctl` matched to the fleet's live Talos version, because a drifted client against a
|
||||
production node is how apply-time surprises happen.
|
||||
|
||||
None of that is exotic. The pattern is the *singularity* of it: ask "where can the fleet be driven from?" and
|
||||
the answer is one hostname, and auditing that answer takes a minute.
|
||||
|
||||
## The bastion is defined by what it refuses
|
||||
|
||||
The keys that live there matter less than the keys that were *refused*. When this box was stood up to take
|
||||
over the operator role from an older machine — before that machine's wipe and rebirth as a GPU node — the
|
||||
login credentials for my agent tooling were deliberately not copied across. Fresh auth on the new box beats a
|
||||
credential that has now existed in two places, because the whole point collapses the moment "one place"
|
||||
becomes "two, temporarily".
|
||||
|
||||
The stand-up also proved the pattern's diagnostic value. Every repo cloned fine — and then a push failed,
|
||||
because the git remotes resolved through SSH host aliases that only existed on the *old* machine. Access had
|
||||
been quietly welded to one particular computer for months, and nothing noticed while that computer was always
|
||||
there. Consolidating forces the question: is this credential something the operator point holds, or something
|
||||
one box happens to have? The fix was the pattern applied properly — a new key generated *on* the bastion,
|
||||
registered once, never exported.
|
||||
|
||||
## The one that costs an evening
|
||||
|
||||
The bastion has to be reachable off-LAN, so it joined the tailnet as a subnet router, advertising its own
|
||||
network. Here's the trap: it also sat *inside* a subnet another router already advertised — and bringing it
|
||||
up with `--accept-routes` made it accept the overlay route *for its own LAN*. Local replies went into the
|
||||
tunnel instead of out the NIC. ARP stayed healthy, every TCP connection died, and SSH to the box that holds
|
||||
all the keys went dark.
|
||||
|
||||
The rule that falls out: a node on the only LAN **advertises routes but never accepts them** — it reaches that
|
||||
network natively. And the recovery is itself a bastion lesson: you get back in *out-of-band*, through the
|
||||
hypervisor console. The control point must sit on infrastructure you can still reach when the network lies.
|
||||
|
||||
## The principle
|
||||
|
||||
A bastion isn't a product you install; it's a property you enforce: **the set of places your keys exist has
|
||||
exactly one member.** Any box can hold the role — a VM, a spare NUC, a cloud instance — and the role can move,
|
||||
the way this one inherited it from a machine headed for a wipe. What never moves casually is the material
|
||||
itself. Ships come and go all day.
|
||||
|
||||
The charts never leave the station.
|
||||
|
||||
*Live as a four-core Debian VM at 10.0.11.11 — one kubeconfig, one talosconfig, one age key, every ops repo —
|
||||
driving a five-cluster Talos fleet, reachable over the tailnet as its second subnet router.*
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
title: "The cert that couldn't see its own proof"
|
||||
date: 2026-07-03
|
||||
summary: "Let's Encrypt DNS-01 posted its proof to the public internet — and then checked for it through the LAN's split-horizon DNS, which will never carry it. The fix is one line: make the self-check look where the CA looks, not where you live."
|
||||
tags: ["tls", "dns", "cert-manager", "lets-encrypt", "split-horizon"]
|
||||
hero: "/blog/cert-couldnt-see-its-proof.webp?v=1"
|
||||
heroAlt: "A sealed structure split by a translucent wall: a glowing proof-token sits in the open on the outside where a distant beam finds it, while an identical vantage on the inside looks toward the same spot and sees nothing."
|
||||
draft: false
|
||||
---
|
||||
|
||||
|
||||
Here's the uncomfortable default: cert-manager checks its own homework **through whatever DNS the cluster hands it**. On a normal network that's fine — the cluster's resolvers see the same internet everyone else does. On a split-horizon network, they don't. Your internal DNS is authoritative for the domain, answers first, and knows nothing about the record you just published to the world.
|
||||
|
||||
So the DNS-01 challenge does something quietly absurd. cert-manager writes the `_acme-challenge` TXT record to Cloudflare — and it lands. `dig @1.1.1.1` returns the token. Let's Encrypt could validate it right now. But before asking the CA to look, cert-manager runs a *self-check*: it resolves the record itself, through cluster DNS, through the Pi-hole that owns `bztmon.org` internally and has no such TXT record. Nothing comes back. So it waits, and checks again, and loops on `"not yet propagated"` — forever.
|
||||
|
||||
It's posting a letter into the public postbox, then walking back inside to check the hallway mail slot for it. The letter is out there. The postman can see it. You're looking in the one box it will never reach — and concluding it hasn't been sent.
|
||||
|
||||
<!-- DIAGRAM: split-horizon DNS-01 — cert-manager writes the TXT to Cloudflare (public path, record live), Let's Encrypt validates via public DNS, but the self-check arrow loops through the internal Pi-hole which returns nothing; the fix arrow reroutes the self-check to 1.1.1.1/8.8.8.8 -->
|
||||
|
||||
|
||||
## Look where the examiner looks
|
||||
|
||||
The fix is one Helm value, and it encodes the whole lesson:
|
||||
|
||||
```yaml
|
||||
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
|
||||
dns01RecursiveNameserversOnly: true
|
||||
```
|
||||
|
||||
That points the self-check at public resolvers — the same vantage Let's Encrypt validates from — instead of the cluster's own view of the world. The moment cert-manager checks the postbox the postman actually collects from, the challenge that had been pending for an age clears in seconds. No record changed. No DNS changed. Only the *observer* moved.
|
||||
|
||||
One footnote if you run default-deny egress: the check now leaves the cluster on port 53, so `world:53` has to be allowed for the cert-manager pods. Here it already was, courtesy of the same policy that lets cloudflared out.
|
||||
|
||||
## Two proofs, one postbox
|
||||
|
||||
The same challenge has a second way to hang, and it looks identical from the outside. Request a certificate for both the wildcard `*.bztmon.org` *and* the apex `bztmon.org`, and Let's Encrypt opens two authorisations — both of which validate at the **same** record name, `_acme-challenge.bztmon.org`, with *different* tokens. cert-manager's Cloudflare solver dutifully writes one, then the other solver overwrites it, then the first writes it back. Two letters fighting over one envelope, each self-check finding the other's contents. Both loop on `"not yet propagated"` — and neither is a propagation problem.
|
||||
|
||||
The fix is to stop asking for both: request the wildcard only, and drop the apex SAN unless you serve the bare domain. This one ate a real evening on robin before the pattern was obvious.
|
||||
|
||||
## A permission failure in a propagation costume
|
||||
|
||||
The third trap doesn't even mention permissions. A Cloudflare API token with `Zone:DNS:Edit` alone *looks* sufficient — it can edit records, what more could a DNS solver want? But cert-manager first has to enumerate zones to find the zone ID, and that needs **`Zone:Zone:Read`**. Without it, the failure surfaces not as `403` but as — you guessed it — `"not yet propagated"`.
|
||||
|
||||
Add Zone:Read to the token at Cloudflare and it unblocks instantly. The token string doesn't change, so no secret rotates; the fix never touches the cluster at all.
|
||||
|
||||
Three different root causes. One identical symptom. That's the real menace of this error message: it names the *last observation* ("I can't see the record"), not the reason.
|
||||
|
||||
## The principle
|
||||
|
||||
A self-check is only worth what its vantage point is worth. If your verifier stands outside — a CA on the public internet, a monitor beyond the LAN, an auditor reading the published copy — then a check that observes from *inside* isn't a weaker version of the truth. It's a different truth, and it can disagree indefinitely.
|
||||
|
||||
So when a system checks its own work before presenting it, ask one question first: does it look from where the judge will look? Check the postbox the postman collects from — not the mail slot in your own hallway.
|
||||
|
||||
*Bit the media-gateway wildcard on elfastc, 2026-07-02 — cert-manager v1.19.5 issuing `*.bztmon.org` via Cloudflare DNS-01, on a fleet where the internal Pi-hole owns the zone and public resolvers hold the proof.*
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: "A backup you haven't restored is a rumour"
|
||||
date: 2026-07-03
|
||||
summary: "Backup jobs report success every night for years — and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught."
|
||||
tags: ["backups", "disaster-recovery", "kubernetes", "restic", "homelab"]
|
||||
hero: "/blog/restore-or-rumour.webp?v=1"
|
||||
heroAlt: "A dim vault with a small tray of retrieved data-crystals under an examination lamp — the proof is what was brought back OUT, not what sits sealed inside."
|
||||
draft: false
|
||||
---
|
||||
|
||||
|
||||
Ask an estate whether it has backups and you'll get a confident yes: the job runs nightly, the dashboard is
|
||||
green, the repository is growing. Ask when anyone last *restored* one and the room goes quiet.
|
||||
|
||||
Here's the uncomfortable framing: an unverified backup is not evidence — it's a rumour. Someone once said
|
||||
the data was safe, the claim got repeated nightly by a cron job, and everyone chose to believe it. The green
|
||||
tick proves a process exited zero. It says nothing about whether the bytes at rest can be turned back into a
|
||||
running application on the worst day of the year.
|
||||
|
||||
## The restore drill is the acceptance test
|
||||
|
||||
When I consolidated the homelab's backups onto the NAS — every cluster's application state streaming into
|
||||
one repository per cluster — the job wasn't "done" when the first snapshot landed. It was done when a
|
||||
scratch pod pulled thousands of files back *out* of the repository, the configs checked intact, and the
|
||||
services proved recoverable from nothing but the backup and its key. Both clusters. Before the old backup
|
||||
path was retired, not after.
|
||||
|
||||
That's the rule worth writing down: **a backup pipeline ships with its restore drill, the same way code
|
||||
ships with its tests.** Not a runbook that describes a restore — an actual rehearsal, on the real
|
||||
repository, with a pass/fail outcome.
|
||||
|
||||
<!-- DIAGRAM: nightly snapshot arrows into a repository; a drill loop pulling a snapshot back OUT into a scratch pod with a green check — the loop is what's highlighted, not the vault -->
|
||||
|
||||
|
||||
## What the drill actually catches
|
||||
|
||||
The failures a drill surfaces are precisely the ones the nightly tick can't see. A repository still locked
|
||||
by a dead process from last week's crashed run — every subsequent "successful" backup queued behind a stale
|
||||
lock. A prune step that had been silently failing. And the one that matters most on the worst day: whether
|
||||
the *key* is where the recovery plan says it is. Every one of those was invisible from the green-tick view,
|
||||
and every one is a nothing-burger to fix on a Tuesday afternoon — and an incident report if discovered
|
||||
during a real recovery.
|
||||
|
||||
There's a quieter design lesson the drill enforces too: back up **state, not just volumes**. A database dump
|
||||
you can restore beats a filesystem snapshot you have to forensically reassemble; the drill makes you notice
|
||||
the difference, because the drill is where you actually try.
|
||||
|
||||
## One key to rule the recovery
|
||||
|
||||
Consolidation concentrates risk on purpose — one NAS, one repository format, one passphrase unlocking every
|
||||
cluster's history. That's a fine trade *if* the key management is treated with the same rigour as the data:
|
||||
the passphrase lives offline, not in any repository it unlocks, and the recovery plan starts from "a bare
|
||||
machine and the key", assuming nothing else survived. A backup encrypted with a key stored next to it is a
|
||||
padlock with the key taped to the shackle.
|
||||
|
||||
## The principle
|
||||
|
||||
Trust receipts, not reports. Any system whose entire purpose is a future emergency — backups, break-glass
|
||||
accounts, failovers, restore paths — is untested by definition on every normal day, so you have to
|
||||
manufacture the test: drill it, on a schedule, with a real pass/fail, and treat a failed drill as a real
|
||||
incident. The nightly green tick is the rumour. The drill is the receipt.
|
||||
|
||||
*Live across the homelab: per-cluster restic repositories on the NAS over SMB, nightly CronJobs, restore
|
||||
drills run before each cutover — thousands of files pulled back and verified per cluster, a stale-lock
|
||||
failure caught and self-healing added, the passphrase held offline.*
|
||||
Reference in New Issue
Block a user