jwright/bztmon-site
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

blog: publish three posts — backup restore drills, the bastion pattern, split-horizon DNS-01 (heroes + diagrams)

Browse Source
This commit is contained in:
Jonathon Wright
2026-07-09 15:02:17 +10:00
parent 2896f89af3
commit 87cfe78ffc
9 changed files with 370 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
public/diagrams/bastion-is-a-pattern.svg
+63
View File
@@ -0,0 +1,63 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="440" viewBox="0 0 1180 440" role="img" aria-label="One bastion control point holds the fleet's keys — one kubeconfig, one talosconfig, one age key, every ops repo — and everything is driven through it: laptops and desktops connect only via the bastion, which reaches out to the five clusters and the git server. The keys never leave the station.">
<rect width="1180" height="440" fill="#ffffff"></rect>
<defs>
<marker id="a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0 0 L10 5 L0 10 z" fill="#0c8fce"></path>
</marker>
<marker id="ag" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0 0 L10 5 L0 10 z" fill="#94a3b8"></path>
</marker>
</defs>
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">One place the keys live &#8212; everything transits it</text>
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">the bastion is a property, not a box: the set of places your keys exist has exactly one member</text>
<!-- operator devices (left) -->
<rect x="48" y="150" width="176" height="60" rx="12" fill="#ffffff" stroke="#94a3b8" stroke-width="1.4"></rect>
<text x="136" y="176" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" font-weight="700" fill="#64748b">laptop</text>
<text x="136" y="196" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10" fill="#94a3b8">holds no keys</text>
<rect x="48" y="230" width="176" height="60" rx="12" fill="#ffffff" stroke="#94a3b8" stroke-width="1.4"></rect>
<text x="136" y="256" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" font-weight="700" fill="#64748b">desktop</text>
<text x="136" y="276" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10" fill="#94a3b8">holds no keys</text>
<!-- the bastion (centre, the station) -->
<rect x="366" y="150" width="240" height="170" rx="16" fill="#7c3aed" fill-opacity="0.05" stroke="#7c3aed" stroke-width="2.4"></rect>
<text x="486" y="184" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="16" font-weight="700" fill="#7c3aed">bastion</text>
<text x="486" y="205" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">the pilot station &#183; 4-core VM</text>
<text x="486" y="232" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">1 kubeconfig &#183; 1 talosconfig</text>
<text x="486" y="252" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">1 age key &#183; every ops repo</text>
<text x="486" y="272" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">toolchain pinned to the fleet</text>
<text x="486" y="300" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" font-weight="700" fill="#7c3aed">keys never leave here</text>
<!-- transit arrows in -->
<path d="M224 180 C300 180 320 210 362 216" fill="none" stroke="#94a3b8" stroke-width="2" marker-end="url(#ag)"></path>
<path d="M224 260 C300 260 320 250 362 244" fill="none" stroke="#94a3b8" stroke-width="2" marker-end="url(#ag)"></path>
<text x="300" y="150" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">connect THROUGH</text>
<!-- the fleet (right, the ships) -->
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">
<rect x="760" y="120" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
<text x="835" y="145" text-anchor="middle">cave</text>
<rect x="940" y="120" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
<text x="1015" y="145" text-anchor="middle">alfred</text>
<rect x="760" y="176" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
<text x="835" y="201" text-anchor="middle">robin</text>
<rect x="940" y="176" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
<text x="1015" y="201" text-anchor="middle">wgirl</text>
<rect x="760" y="232" width="150" height="40" rx="10" fill="#ffffff" stroke="#0c8fce" stroke-width="1.5"></rect>
<text x="835" y="257" text-anchor="middle">elfastc</text>
<rect x="940" y="232" width="150" height="40" rx="10" fill="#ffffff" stroke="#7c3aed" stroke-width="1.5"></rect>
<text x="1015" y="257" text-anchor="middle" fill="#7c3aed">git server</text>
</g>
<!-- beams out to the fleet -->
<g fill="none" stroke="#0c8fce" stroke-width="2">
<path d="M606 200 C680 180 700 150 756 140" marker-end="url(#a)"></path>
<path d="M606 214 C700 210 720 196 936 196" marker-end="url(#a)"></path>
<path d="M606 232 C680 250 700 252 756 252" marker-end="url(#a)"></path>
</g>
<path d="M606 244 C740 300 860 270 936 252" fill="none" stroke="#7c3aed" stroke-width="2" marker-end="url(#a)"></path>
<text x="690" y="330" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">driven from one hostname &#183; auditing &#8220;where from?&#8221; takes a minute</text>
<text x="60" y="404" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" fill="#64748b">ships come and go &#8212; the charts never leave the station</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 5.9 KiB

public/diagrams/cert-couldnt-see-its-proof.svg
+54
View File
@@ -0,0 +1,54 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="450" viewBox="0 0 1180 450" role="img" aria-label="cert-manager writes the ACME challenge TXT record to Cloudflare, where it goes live on the public internet and Let's Encrypt validates it successfully. But its own self-check resolves through the cluster's split-horizon DNS — the internal Pi-hole that owns the zone and has no such record — so it loops forever on not-yet-propagated. The fix points the self-check at public resolvers, the same vantage the CA uses.">
<rect width="1180" height="450" fill="#ffffff"></rect>
<defs>
<marker id="g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0 L10 5 L0 10 z" fill="#0c8fce"></path></marker>
<marker id="r" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0 L10 5 L0 10 z" fill="#c026d3"></path></marker>
<marker id="v" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0 L10 5 L0 10 z" fill="#7c3aed"></path></marker>
</defs>
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">The proof is out there &#8212; the self-check looks inside</text>
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">a check is only worth its vantage point: verify from where the judge stands, not from where you live</text>
<!-- cert-manager -->
<rect x="52" y="176" width="196" height="110" rx="14" fill="#ffffff" stroke="#0e1726" stroke-width="1.6"></rect>
<text x="150" y="212" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#0e1726">cert-manager</text>
<text x="150" y="238" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">writes the TXT, then</text>
<text x="150" y="256" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">self-checks before</text>
<text x="150" y="274" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">calling the CA</text>
<!-- write to Cloudflare (works) -->
<path d="M248 200 H432" fill="none" stroke="#0c8fce" stroke-width="2.4" marker-end="url(#g)"></path>
<text x="340" y="190" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0c8fce">write _acme-challenge TXT</text>
<rect x="440" y="150" width="220" height="96" rx="14" fill="#0c8fce" fill-opacity="0.05" stroke="#0c8fce" stroke-width="2"></rect>
<text x="550" y="182" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14.5" font-weight="700" fill="#0c8fce">Cloudflare (public)</text>
<text x="550" y="206" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">record is LIVE</text>
<text x="550" y="226" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">dig @1.1.1.1 returns the token</text>
<!-- Let's Encrypt validates from public -->
<rect x="760" y="150" width="200" height="96" rx="14" fill="#ffffff" stroke="#7c3aed" stroke-width="1.8"></rect>
<text x="860" y="182" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14.5" font-weight="700" fill="#7c3aed">Let&#8217;s Encrypt</text>
<text x="860" y="206" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">validates via public DNS</text>
<text x="860" y="226" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">would pass right now</text>
<path d="M660 198 H752" fill="none" stroke="#7c3aed" stroke-width="2" marker-end="url(#v)"></path>
<!-- the broken self-check: through the Pi-hole (magenta) -->
<rect x="440" y="300" width="220" height="100" rx="14" fill="#c026d3" fill-opacity="0.05" stroke="#c026d3" stroke-width="2"></rect>
<text x="550" y="332" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14" font-weight="700" fill="#c026d3">internal Pi-hole</text>
<text x="550" y="356" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">owns bztmon.org internally</text>
<text x="550" y="376" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">no such TXT &#8594; returns nothing</text>
<path d="M150 286 C150 350 300 360 432 356" fill="none" stroke="#c026d3" stroke-width="2.4" marker-end="url(#r)"></path>
<text x="290" y="380" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#c026d3">self-check (cluster DNS)</text>
<path d="M470 360 C428 348 424 322 448 310" fill="none" stroke="#c026d3" stroke-width="2" stroke-dasharray="5 5" marker-end="url(#r)"></path>
<text x="550" y="416" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#c026d3">loops forever: &#8220;not yet propagated&#8221;</text>
<!-- the fix -->
<rect x="760" y="300" width="360" height="100" rx="14" fill="#7c3aed" fill-opacity="0.05" stroke="#7c3aed" stroke-width="2"></rect>
<text x="784" y="330" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" font-weight="700" fill="#7c3aed">the fix &#8212; move the observer, not the record</text>
<text x="784" y="356" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">dns01RecursiveNameservers: 1.1.1.1,8.8.8.8</text>
<text x="784" y="376" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">dns01RecursiveNameserversOnly: true</text>
<path d="M660 350 C700 350 720 350 752 350" fill="none" stroke="#7c3aed" stroke-width="2" stroke-dasharray="5 5" marker-end="url(#v)"></path>
<text x="60" y="430" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" fill="#64748b">check the postbox the postman collects from &#8212; not the mail slot in your own hallway</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 6.5 KiB

public/diagrams/restore-or-rumour.svg
+55
View File
@@ -0,0 +1,55 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="440" viewBox="0 0 1180 440" role="img" aria-label="Nightly backups stream from the clusters into a repository on the NAS and every job reports green; the highlighted drill loop pulls a snapshot back out into a scratch pod, verifies files and configs, and produces a pass or fail — the loop out of the vault is the proof, not the data going in.">
<rect width="1180" height="440" fill="#ffffff"></rect>
<defs>
<marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0 0 L10 5 L0 10 z" fill="#94a3b8"></path>
</marker>
<marker id="arrC" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0 0 L10 5 L0 10 z" fill="#0c8fce"></path>
</marker>
</defs>
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">The drill loop is the receipt</text>
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">data flowing in happens every night &#8212; only data pulled back OUT proves the backup is real</text>
<!-- clusters -->
<rect x="56" y="150" width="180" height="84" rx="14" fill="#ffffff" stroke="#0e1726" stroke-width="1.6"></rect>
<text x="146" y="184" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14.5" font-weight="700" fill="#0e1726">cluster apps</text>
<text x="146" y="208" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">PVCs &#183; dumps &#183; state</text>
<rect x="56" y="250" width="180" height="60" rx="14" fill="#ffffff" stroke="#94a3b8" stroke-width="1.4"></rect>
<text x="146" y="278" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" font-weight="700" fill="#64748b">nightly CronJob</text>
<text x="146" y="296" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">exit 0 &#8594; green tick</text>
<!-- repository vault -->
<rect x="380" y="150" width="230" height="160" rx="16" fill="#7c3aed" fill-opacity="0.05" stroke="#7c3aed" stroke-width="2"></rect>
<text x="495" y="186" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15.5" font-weight="700" fill="#7c3aed">repository</text>
<text x="495" y="210" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">restic on the NAS &#183; encrypted</text>
<text x="495" y="230" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">key held offline &#8212; never</text>
<text x="495" y="246" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">stored beside the data</text>
<text x="495" y="284" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">a green tick ends here</text>
<!-- backup arrows (routine, grey) -->
<path d="M236 192 H372" fill="none" stroke="#94a3b8" stroke-width="2" marker-end="url(#arr)"></path>
<text x="304" y="182" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">snapshot in</text>
<path d="M236 280 C300 280 320 250 372 236" fill="none" stroke="#94a3b8" stroke-width="2" marker-end="url(#arr)"></path>
<!-- THE DRILL LOOP (highlighted, cyan) -->
<rect x="700" y="150" width="210" height="110" rx="14" fill="#0c8fce" fill-opacity="0.05" stroke="#0c8fce" stroke-width="2.2"></rect>
<text x="805" y="184" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="14.5" font-weight="700" fill="#0c8fce">scratch pod</text>
<text x="805" y="208" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#64748b">restore into nothing</text>
<text x="805" y="228" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">files pulled &#183; configs read</text>
<text x="805" y="244" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">services prove they start</text>
<rect x="980" y="164" width="140" height="82" rx="14" fill="#ffffff" stroke="#0c8fce" stroke-width="2"></rect>
<text x="1050" y="199" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="26" font-weight="700" fill="#0c8fce">&#10003;</text>
<text x="1050" y="228" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0e1726">pass / fail</text>
<path d="M610 205 H692" fill="none" stroke="#0c8fce" stroke-width="3" marker-end="url(#arrC)"></path>
<text x="651" y="195" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" font-weight="700" fill="#0c8fce">snapshot OUT</text>
<path d="M910 205 H972" fill="none" stroke="#0c8fce" stroke-width="3" marker-end="url(#arrC)"></path>
<path d="M805 260 C805 330 560 330 500 314" fill="none" stroke="#0c8fce" stroke-width="2" stroke-dasharray="6 5" marker-end="url(#arrC)"></path>
<text x="672" y="342" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="10.5" fill="#0c8fce">drill on a schedule &#183; failed drill = real incident</text>
<text x="60" y="404" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13" fill="#64748b">the vault is not the proof &#8212; the loop out of it is</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 5.9 KiB