blog: publish three posts — backup restore drills, the bastion pattern, split-horizon DNS-01 (heroes + diagrams)

2026-07-09 15:02:17 +10:00
parent 2896f89af3
commit 87cfe78ffc
9 changed files with 370 additions and 0 deletions
@@ -0,0 +1,63 @@
 <svg xmlns="http://www.w3.org/2000/svg" width="1180" height="440" viewBox="0 0 1180 440" role="img" aria-label="One bastion control point holds the fleet's keys — one kubeconfig, one talosconfig, one age key, every ops repo — and everything is driven through it: laptops and desktops connect only via the bastion, which reaches out to the five clusters and the git server. The keys never leave the station.">
  <rect width="1180" height="440" fill="#ffffff"></rect>
  <defs>
    <marker id="a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="#0c8fce"></path>
    </marker>
    <marker id="ag" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="#94a3b8"></path>
    </marker>
  </defs>
  <text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">One place the keys live &#8212; everything transits it</text>
  <text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">the bastion is a property, not a box: the set of places your keys exist has exactly one member</text>
  <!-- operator devices (left) -->
  <rect x="48" y="150" width="176" height="60" rx="12" fill="#ffffff" stroke="#94a3b8" stroke-width="1.4"></rect>
  <text x="136" y="176" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" font-weight="700" fill="#64748b">laptop</text>
  <text x="136" y="196" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10" fill="#94a3b8">holds no keys</text>
  <rect x="48" y="230" width="176" height="60" rx="12" fill="#ffffff" stroke="#94a3b8" stroke-width="1.4"></rect>
  <text x="136" y="256" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" font-weight="700" fill="#64748b">desktop</text>
  <text x="136" y="276" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10" fill="#94a3b8">holds no keys</text>
  <!-- the bastion (centre, the station) -->
  <rect x="366" y="150" width="240" height="170" rx="16" fill="#7c3aed" fill-opacity="0.05" stroke="#7c3aed" stroke-width="2.4"></rect>
  <text x="486" y="184" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="16" font-weight="700" fill="#7c3aed">bastion</text>
  <text x="486" y="205" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">the pilot station &#183; 4-core VM</text>
  <text x="486" y="232" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">1 kubeconfig &#183; 1 talosconfig</text>
  <text x="486" y="252" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">1 age key &#183; every ops repo</text>
  <text x="486" y="272" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">toolchain pinned to the fleet</text>
  <text x="486" y="300" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" font-weight="700" fill="#7c3aed">keys never leave here</text>
  <!-- transit arrows in -->
  <path d="M224 180 C300 180 320 210 362 216" fill="none" stroke="#94a3b8" stroke-width="2" marker-end="url(#ag)"></path>
  <path d="M224 260 C300 260 320 250 362 244" fill="none" stroke="#94a3b8" stroke-width="2" marker-end="url(#ag)"></path>
  <text x="300" y="150" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">connect THROUGH</text>
  <!-- the fleet (right, the ships) -->
  <g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">
    <rect x="760" y="120" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
    <text x="835" y="145" text-anchor="middle">cave</text>
    <rect x="940" y="120" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
    <text x="1015" y="145" text-anchor="middle">alfred</text>
    <rect x="760" y="176" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
    <text x="835" y="201" text-anchor="middle">robin</text>
    <rect x="940" y="176" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
    <text x="1015" y="201" text-anchor="middle">wgirl</text>
    <rect x="760" y="232" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
    <text x="835" y="257" text-anchor="middle">elfastc</text>
    <rect x="940" y="232" width="150" height="40" rx="10" fill="#ffffff" stroke="#7c3aed" stroke-width="1.5"></rect>
    <text x="1015" y="257" text-anchor="middle" fill="#7c3aed">git server</text>
  </g>
  <!-- beams out to the fleet -->
  <g fill="none" stroke="#0c8fce" stroke-width="2">
    <path d="M606 200 C680 180 700 150 756 140" marker-end="url(#a)"></path>
    <path d="M606 214 C700 210 720 196 936 196" marker-end="url(#a)"></path>
    <path d="M606 232 C680 250 700 252 756 252" marker-end="url(#a)"></path>
  </g>
  <path d="M606 244 C740 300 860 270 936 252" fill="none" stroke="#7c3aed" stroke-width="2" marker-end="url(#a)"></path>
  <text x="690" y="330" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">driven from one hostname &#183; auditing &#8220;where from?&#8221; takes a minute</text>
  <text x="60" y="404" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" fill="#64748b">ships come and go &#8212; the charts never leave the station</text>
 </svg>
@@ -0,0 +1,54 @@
 <svg xmlns="http://www.w3.org/2000/svg" width="1180" height="450" viewBox="0 0 1180 450" role="img" aria-label="cert-manager writes the ACME challenge TXT record to Cloudflare, where it goes live on the public internet and Let's Encrypt validates it successfully. But its own self-check resolves through the cluster's split-horizon DNS — the internal Pi-hole that owns the zone and has no such record — so it loops forever on not-yet-propagated. The fix points the self-check at public resolvers, the same vantage the CA uses.">
  <rect width="1180" height="450" fill="#ffffff"></rect>
  <defs>
    <marker id="g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0 L10 5 L0 10 z" fill="#0c8fce"></path></marker>
    <marker id="r" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0 L10 5 L0 10 z" fill="#c026d3"></path></marker>
    <marker id="v" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0 L10 5 L0 10 z" fill="#7c3aed"></path></marker>
  </defs>
  <text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">The proof is out there &#8212; the self-check looks inside</text>
  <text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">a check is only worth its vantage point: verify from where the judge stands, not from where you live</text>
  <!-- cert-manager -->
  <rect x="52" y="176" width="196" height="110" rx="14" fill="#ffffff" stroke="#0e1726" stroke-width="1.6"></rect>
  <text x="150" y="212" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#0e1726">cert-manager</text>
  <text x="150" y="238" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">writes the TXT, then</text>
  <text x="150" y="256" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">self-checks before</text>
  <text x="150" y="274" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">calling the CA</text>
  <!-- write to Cloudflare (works) -->
  <path d="M248 200 H432" fill="none" stroke="#0c8fce" stroke-width="2.4" marker-end="url(#g)"></path>
  <text x="340" y="190" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0c8fce">write _acme-challenge TXT</text>
  <rect x="440" y="150" width="220" height="96" rx="14" fill="#0c8fce" fill-opacity="0.05" stroke="#0c8fce" stroke-width="2"></rect>
  <text x="550" y="182" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14.5" font-weight="700" fill="#0c8fce">Cloudflare (public)</text>
  <text x="550" y="206" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">record is LIVE</text>
  <text x="550" y="226" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">dig @1.1.1.1 returns the token</text>
  <!-- Let's Encrypt validates from public -->
  <rect x="760" y="150" width="200" height="96" rx="14" fill="#ffffff" stroke="#7c3aed" stroke-width="1.8"></rect>
  <text x="860" y="182" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14.5" font-weight="700" fill="#7c3aed">Let&#8217;s Encrypt</text>
  <text x="860" y="206" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">validates via public DNS</text>
  <text x="860" y="226" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">would pass right now</text>
  <path d="M660 198 H752" fill="none" stroke="#7c3aed" stroke-width="2" marker-end="url(#v)"></path>
  <!-- the broken self-check: through the Pi-hole (magenta) -->
  <rect x="440" y="300" width="220" height="100" rx="14" fill="#c026d3" fill-opacity="0.05" stroke="#c026d3" stroke-width="2"></rect>
  <text x="550" y="332" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14" font-weight="700" fill="#c026d3">internal Pi-hole</text>
  <text x="550" y="356" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">owns bztmon.org internally</text>
  <text x="550" y="376" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">no such TXT &#8594; returns nothing</text>
  <path d="M150 286 C150 350 300 360 432 356" fill="none" stroke="#c026d3" stroke-width="2.4" marker-end="url(#r)"></path>
  <text x="290" y="380" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#c026d3">self-check (cluster DNS)</text>
  <path d="M470 360 C428 348 424 322 448 310" fill="none" stroke="#c026d3" stroke-width="2" stroke-dasharray="5 5" marker-end="url(#r)"></path>
  <text x="550" y="416" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#c026d3">loops forever: &#8220;not yet propagated&#8221;</text>
  <!-- the fix -->
  <rect x="760" y="300" width="360" height="100" rx="14" fill="#7c3aed" fill-opacity="0.05" stroke="#7c3aed" stroke-width="2"></rect>
  <text x="784" y="330" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" font-weight="700" fill="#7c3aed">the fix &#8212; move the observer, not the record</text>
  <text x="784" y="356" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">dns01RecursiveNameservers: 1.1.1.1,8.8.8.8</text>
  <text x="784" y="376" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">dns01RecursiveNameserversOnly: true</text>
  <path d="M660 350 C700 350 720 350 752 350" fill="none" stroke="#7c3aed" stroke-width="2" stroke-dasharray="5 5" marker-end="url(#v)"></path>
  <text x="60" y="430" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" fill="#64748b">check the postbox the postman collects from &#8212; not the mail slot in your own hallway</text>
 </svg>
@@ -0,0 +1,55 @@
 <svg xmlns="http://www.w3.org/2000/svg" width="1180" height="440" viewBox="0 0 1180 440" role="img" aria-label="Nightly backups stream from the clusters into a repository on the NAS and every job reports green; the highlighted drill loop pulls a snapshot back out into a scratch pod, verifies files and configs, and produces a pass or fail — the loop out of the vault is the proof, not the data going in.">
  <rect width="1180" height="440" fill="#ffffff"></rect>
  <defs>
    <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="#94a3b8"></path>
    </marker>
    <marker id="arrC" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="#0c8fce"></path>
    </marker>
  </defs>
  <text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">The drill loop is the receipt</text>
  <text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">data flowing in happens every night &#8212; only data pulled back OUT proves the backup is real</text>
  <!-- clusters -->
  <rect x="56" y="150" width="180" height="84" rx="14" fill="#ffffff" stroke="#0e1726" stroke-width="1.6"></rect>
  <text x="146" y="184" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14.5" font-weight="700" fill="#0e1726">cluster apps</text>
  <text x="146" y="208" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">PVCs &#183; dumps &#183; state</text>
  <rect x="56" y="250" width="180" height="60" rx="14" fill="#ffffff" stroke="#94a3b8" stroke-width="1.4"></rect>
  <text x="146" y="278" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" font-weight="700" fill="#64748b">nightly CronJob</text>
  <text x="146" y="296" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">exit 0 &#8594; green tick</text>
  <!-- repository vault -->
  <rect x="380" y="150" width="230" height="160" rx="16" fill="#7c3aed" fill-opacity="0.05" stroke="#7c3aed" stroke-width="2"></rect>
  <text x="495" y="186" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15.5" font-weight="700" fill="#7c3aed">repository</text>
  <text x="495" y="210" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">restic on the NAS &#183; encrypted</text>
  <text x="495" y="230" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">key held offline &#8212; never</text>
  <text x="495" y="246" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">stored beside the data</text>
  <text x="495" y="284" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">a green tick ends here</text>
  <!-- backup arrows (routine, grey) -->
  <path d="M236 192 H372" fill="none" stroke="#94a3b8" stroke-width="2" marker-end="url(#arr)"></path>
  <text x="304" y="182" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">snapshot in</text>
  <path d="M236 280 C300 280 320 250 372 236" fill="none" stroke="#94a3b8" stroke-width="2" marker-end="url(#arr)"></path>
  <!-- THE DRILL LOOP (highlighted, cyan) -->
  <rect x="700" y="150" width="210" height="110" rx="14" fill="#0c8fce" fill-opacity="0.05" stroke="#0c8fce" stroke-width="2.2"></rect>
  <text x="805" y="184" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14.5" font-weight="700" fill="#0c8fce">scratch pod</text>
  <text x="805" y="208" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">restore into nothing</text>
  <text x="805" y="228" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">files pulled &#183; configs read</text>
  <text x="805" y="244" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">services prove they start</text>
  <rect x="980" y="164" width="140" height="82" rx="14" fill="#ffffff" stroke="#0c8fce" stroke-width="2"></rect>
  <text x="1050" y="199" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="26" font-weight="700" fill="#0c8fce">&#10003;</text>
  <text x="1050" y="228" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">pass / fail</text>
  <path d="M610 205 H692" fill="none" stroke="#0c8fce" stroke-width="3" marker-end="url(#arrC)"></path>
  <text x="651" y="195" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" font-weight="700" fill="#0c8fce">snapshot OUT</text>
  <path d="M910 205 H972" fill="none" stroke="#0c8fce" stroke-width="3" marker-end="url(#arrC)"></path>
  <path d="M805 260 C805 330 560 330 500 314" fill="none" stroke="#0c8fce" stroke-width="2" stroke-dasharray="6 5" marker-end="url(#arrC)"></path>
  <text x="672" y="342" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0c8fce">drill on a schedule &#183; failed drill = real incident</text>
  <text x="60" y="404" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" fill="#64748b">the vault is not the proof &#8212; the loop out of it is</text>
 </svg>
@@ -0,0 +1,79 @@
 ---
 title: "The bastion is a pattern, not a box"
 date: 2026-07-03
 summary: "Admin credentials accumulate on whatever machine you happen to work from, until your daily driver is the real control plane of the fleet. A bastion fixes that — not by being special hardware, but by being the one place the keys live and the discipline that they never leave it."
 tags: ["security", "operations", "homelab", "tailscale", "access"]
 hero: "/blog/bastion-is-a-pattern.webp?v=1"
 heroAlt: "A lone illuminated harbour pilot station on a dark sea, thin beams of cyan and magenta light reaching out to distant vessels — the charts and keys stay at the station, the ships only transit past it."
 draft: false
 ---
 Here's the uncomfortable default: your fleet's admin credentials live wherever you last needed them. A
 kubeconfig on the laptop, another copy on the desktop, an age key pasted onto a third machine "just for one
 decrypt", SSH keys sprouting wherever a clone was convenient. Nobody decided this. It accumulates.
 Which means your **daily driver is the real control plane** — the machine that browses the web, opens email
 attachments and runs whatever you installed last week is also the machine that can wipe a cluster. Lose it,
 rebuild it, or compromise it, and you've done all three to the fleet.
 The inversion is old and it still works: pick *one* point, put every key there, and make the rule that they
 never leave it. Not a hardened appliance. A pattern.
 <!-- DIAGRAM: one bastion node holding keys/contexts/repos, lines out to the clusters + git; laptops connect only THROUGH it -->
 ![One operator control point holding the keys, with everything driven through it](/diagrams/bastion-is-a-pattern.svg?v=1)
 ## Everything transits, nothing homes anywhere else
 Think of a harbour pilot station. Every ship entering the harbour is guided by a pilot, but the charts and the
 local knowledge never board the ships — they stay at the station, and the vessels transit past it. That's the
 whole design: the dangerous knowledge has exactly one home, and traffic comes *to* it.
 Mine is nothing special on purpose: a small Debian VM — four cores, no GPU passthrough, because a bastion needs
 none. What makes it the bastion is what it holds. One kubeconfig with a context for every cluster. The
 talosconfig for the OS layer beneath them. The SOPS age key that decrypts the fleet's secrets, sitting in
 exactly one place at `~/.config/sops/age/keys.txt`. A checkout of every ops repo. And a toolchain pinned
 deliberately — `talosctl` matched to the fleet's live Talos version, because a drifted client against a
 production node is how apply-time surprises happen.
 None of that is exotic. The pattern is the *singularity* of it: ask "where can the fleet be driven from?" and
 the answer is one hostname, and auditing that answer takes a minute.
 ## The bastion is defined by what it refuses
 The keys that live there matter less than the keys that were *refused*. When this box was stood up to take
 over the operator role from an older machine — before that machine's wipe and rebirth as a GPU node — the
 login credentials for my agent tooling were deliberately not copied across. Fresh auth on the new box beats a
 credential that has now existed in two places, because the whole point collapses the moment "one place"
 becomes "two, temporarily".
 The stand-up also proved the pattern's diagnostic value. Every repo cloned fine — and then a push failed,
 because the git remotes resolved through SSH host aliases that only existed on the *old* machine. Access had
 been quietly welded to one particular computer for months, and nothing noticed while that computer was always
 there. Consolidating forces the question: is this credential something the operator point holds, or something
 one box happens to have? The fix was the pattern applied properly — a new key generated *on* the bastion,
 registered once, never exported.
 ## The one that costs an evening
 The bastion has to be reachable off-LAN, so it joined the tailnet as a subnet router, advertising its own
 network. Here's the trap: it also sat *inside* a subnet another router already advertised — and bringing it
 up with `--accept-routes` made it accept the overlay route *for its own LAN*. Local replies went into the
 tunnel instead of out the NIC. ARP stayed healthy, every TCP connection died, and SSH to the box that holds
 all the keys went dark.
 The rule that falls out: a node on the only LAN **advertises routes but never accepts them** — it reaches that
 network natively. And the recovery is itself a bastion lesson: you get back in *out-of-band*, through the
 hypervisor console. The control point must sit on infrastructure you can still reach when the network lies.
 ## The principle
 A bastion isn't a product you install; it's a property you enforce: **the set of places your keys exist has
 exactly one member.** Any box can hold the role — a VM, a spare NUC, a cloud instance — and the role can move,
 the way this one inherited it from a machine headed for a wipe. What never moves casually is the material
 itself. Ships come and go all day.
 The charts never leave the station.
 *Live as a four-core Debian VM at 10.0.11.11 — one kubeconfig, one talosconfig, one age key, every ops repo —
 driving a five-cluster Talos fleet, reachable over the tailnet as its second subnet router.*
@@ -0,0 +1,54 @@
 ---
 title: "The cert that couldn't see its own proof"
 date: 2026-07-03
 summary: "Let's Encrypt DNS-01 posted its proof to the public internet — and then checked for it through the LAN's split-horizon DNS, which will never carry it. The fix is one line: make the self-check look where the CA looks, not where you live."
 tags: ["tls", "dns", "cert-manager", "lets-encrypt", "split-horizon"]
 hero: "/blog/cert-couldnt-see-its-proof.webp?v=1"
 heroAlt: "A sealed structure split by a translucent wall: a glowing proof-token sits in the open on the outside where a distant beam finds it, while an identical vantage on the inside looks toward the same spot and sees nothing."
 draft: false
 ---
 Here's the uncomfortable default: cert-manager checks its own homework **through whatever DNS the cluster hands it**. On a normal network that's fine — the cluster's resolvers see the same internet everyone else does. On a split-horizon network, they don't. Your internal DNS is authoritative for the domain, answers first, and knows nothing about the record you just published to the world.
 So the DNS-01 challenge does something quietly absurd. cert-manager writes the `_acme-challenge` TXT record to Cloudflare — and it lands. `dig @1.1.1.1` returns the token. Let's Encrypt could validate it right now. But before asking the CA to look, cert-manager runs a *self-check*: it resolves the record itself, through cluster DNS, through the Pi-hole that owns `bztmon.org` internally and has no such TXT record. Nothing comes back. So it waits, and checks again, and loops on `"not yet propagated"` — forever.
 It's posting a letter into the public postbox, then walking back inside to check the hallway mail slot for it. The letter is out there. The postman can see it. You're looking in the one box it will never reach — and concluding it hasn't been sent.
 <!-- DIAGRAM: split-horizon DNS-01 — cert-manager writes the TXT to Cloudflare (public path, record live), Let's Encrypt validates via public DNS, but the self-check arrow loops through the internal Pi-hole which returns nothing; the fix arrow reroutes the self-check to 1.1.1.1/8.8.8.8 -->
 ![The self-check resolving through the wrong horizon while the CA sees the record](/diagrams/cert-couldnt-see-its-proof.svg?v=1)
 ## Look where the examiner looks
 The fix is one Helm value, and it encodes the whole lesson:
 ```yaml
 dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
 dns01RecursiveNameserversOnly: true
 ```
 That points the self-check at public resolvers — the same vantage Let's Encrypt validates from — instead of the cluster's own view of the world. The moment cert-manager checks the postbox the postman actually collects from, the challenge that had been pending for an age clears in seconds. No record changed. No DNS changed. Only the *observer* moved.
 One footnote if you run default-deny egress: the check now leaves the cluster on port 53, so `world:53` has to be allowed for the cert-manager pods. Here it already was, courtesy of the same policy that lets cloudflared out.
 ## Two proofs, one postbox
 The same challenge has a second way to hang, and it looks identical from the outside. Request a certificate for both the wildcard `*.bztmon.org` *and* the apex `bztmon.org`, and Let's Encrypt opens two authorisations — both of which validate at the **same** record name, `_acme-challenge.bztmon.org`, with *different* tokens. cert-manager's Cloudflare solver dutifully writes one, then the other solver overwrites it, then the first writes it back. Two letters fighting over one envelope, each self-check finding the other's contents. Both loop on `"not yet propagated"` — and neither is a propagation problem.
 The fix is to stop asking for both: request the wildcard only, and drop the apex SAN unless you serve the bare domain. This one ate a real evening on robin before the pattern was obvious.
 ## A permission failure in a propagation costume
 The third trap doesn't even mention permissions. A Cloudflare API token with `Zone:DNS:Edit` alone *looks* sufficient — it can edit records, what more could a DNS solver want? But cert-manager first has to enumerate zones to find the zone ID, and that needs **`Zone:Zone:Read`**. Without it, the failure surfaces not as `403` but as — you guessed it — `"not yet propagated"`.
 Add Zone:Read to the token at Cloudflare and it unblocks instantly. The token string doesn't change, so no secret rotates; the fix never touches the cluster at all.
 Three different root causes. One identical symptom. That's the real menace of this error message: it names the *last observation* ("I can't see the record"), not the reason.
 ## The principle
 A self-check is only worth what its vantage point is worth. If your verifier stands outside — a CA on the public internet, a monitor beyond the LAN, an auditor reading the published copy — then a check that observes from *inside* isn't a weaker version of the truth. It's a different truth, and it can disagree indefinitely.
 So when a system checks its own work before presenting it, ask one question first: does it look from where the judge will look? Check the postbox the postman collects from — not the mail slot in your own hallway.
 *Bit the media-gateway wildcard on elfastc, 2026-07-02 — cert-manager v1.19.5 issuing `*.bztmon.org` via Cloudflare DNS-01, on a fleet where the internal Pi-hole owns the zone and public resolvers hold the proof.*
@@ -0,0 +1,65 @@
 ---
 title: "A backup you haven't restored is a rumour"
 date: 2026-07-03
 summary: "Backup jobs report success every night for years — and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught."
 tags: ["backups", "disaster-recovery", "kubernetes", "restic", "homelab"]
 hero: "/blog/restore-or-rumour.webp?v=1"
 heroAlt: "A dim vault with a small tray of retrieved data-crystals under an examination lamp — the proof is what was brought back OUT, not what sits sealed inside."
 draft: false
 ---
 Ask an estate whether it has backups and you'll get a confident yes: the job runs nightly, the dashboard is
 green, the repository is growing. Ask when anyone last *restored* one and the room goes quiet.
 Here's the uncomfortable framing: an unverified backup is not evidence — it's a rumour. Someone once said
 the data was safe, the claim got repeated nightly by a cron job, and everyone chose to believe it. The green
 tick proves a process exited zero. It says nothing about whether the bytes at rest can be turned back into a
 running application on the worst day of the year.
 ## The restore drill is the acceptance test
 When I consolidated the homelab's backups onto the NAS — every cluster's application state streaming into
 one repository per cluster — the job wasn't "done" when the first snapshot landed. It was done when a
 scratch pod pulled thousands of files back *out* of the repository, the configs checked intact, and the
 services proved recoverable from nothing but the backup and its key. Both clusters. Before the old backup
 path was retired, not after.
 That's the rule worth writing down: **a backup pipeline ships with its restore drill, the same way code
 ships with its tests.** Not a runbook that describes a restore — an actual rehearsal, on the real
 repository, with a pass/fail outcome.
 <!-- DIAGRAM: nightly snapshot arrows into a repository; a drill loop pulling a snapshot back OUT into a scratch pod with a green check — the loop is what's highlighted, not the vault -->
 ![The drill loop: data out of the vault is the proof, not data in](/diagrams/restore-or-rumour.svg?v=1)
 ## What the drill actually catches
 The failures a drill surfaces are precisely the ones the nightly tick can't see. A repository still locked
 by a dead process from last week's crashed run — every subsequent "successful" backup queued behind a stale
 lock. A prune step that had been silently failing. And the one that matters most on the worst day: whether
 the *key* is where the recovery plan says it is. Every one of those was invisible from the green-tick view,
 and every one is a nothing-burger to fix on a Tuesday afternoon — and an incident report if discovered
 during a real recovery.
 There's a quieter design lesson the drill enforces too: back up **state, not just volumes**. A database dump
 you can restore beats a filesystem snapshot you have to forensically reassemble; the drill makes you notice
 the difference, because the drill is where you actually try.
 ## One key to rule the recovery
 Consolidation concentrates risk on purpose — one NAS, one repository format, one passphrase unlocking every
 cluster's history. That's a fine trade *if* the key management is treated with the same rigour as the data:
 the passphrase lives offline, not in any repository it unlocks, and the recovery plan starts from "a bare
 machine and the key", assuming nothing else survived. A backup encrypted with a key stored next to it is a
 padlock with the key taped to the shackle.
 ## The principle
 Trust receipts, not reports. Any system whose entire purpose is a future emergency — backups, break-glass
 accounts, failovers, restore paths — is untested by definition on every normal day, so you have to
 manufacture the test: drill it, on a schedule, with a real pass/fail, and treat a failed drill as a real
 incident. The nightly green tick is the rumour. The drill is the receipt.
 *Live across the homelab: per-cluster restic repositories on the NAS over SMB, nightly CronJobs, restore
 drills run before each cutover — thousands of files pulled back and verified per cluster, a stale-lock
 failure caught and self-healing added, the passphrase held offline.*