blog: publish three posts — backup restore drills, the bastion pattern, split-horizon DNS-01 (heroes + diagrams)

src/content/blog/restore-or-rumour.md

@@ -0,0 +1,65 @@
+---
+title: "A backup you haven't restored is a rumour"
+date: 2026-07-03
+summary: "Backup jobs report success every night for years — and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught."
+tags: ["backups", "disaster-recovery", "kubernetes", "restic", "homelab"]
+hero: "/blog/restore-or-rumour.webp?v=1"
+heroAlt: "A dim vault with a small tray of retrieved data-crystals under an examination lamp — the proof is what was brought back OUT, not what sits sealed inside."
+draft: false
+---
+
+
+Ask an estate whether it has backups and you'll get a confident yes: the job runs nightly, the dashboard is
+green, the repository is growing. Ask when anyone last *restored* one and the room goes quiet.
+
+Here's the uncomfortable framing: an unverified backup is not evidence — it's a rumour. Someone once said
+the data was safe, the claim got repeated nightly by a cron job, and everyone chose to believe it. The green
+tick proves a process exited zero. It says nothing about whether the bytes at rest can be turned back into a
+running application on the worst day of the year.
+
+## The restore drill is the acceptance test
+
+When I consolidated the homelab's backups onto the NAS — every cluster's application state streaming into
+one repository per cluster — the job wasn't "done" when the first snapshot landed. It was done when a
+scratch pod pulled thousands of files back *out* of the repository, the configs checked intact, and the
+services proved recoverable from nothing but the backup and its key. Both clusters. Before the old backup
+path was retired, not after.
+
+That's the rule worth writing down: **a backup pipeline ships with its restore drill, the same way code
+ships with its tests.** Not a runbook that describes a restore — an actual rehearsal, on the real
+repository, with a pass/fail outcome.
+
+<!-- DIAGRAM: nightly snapshot arrows into a repository; a drill loop pulling a snapshot back OUT into a scratch pod with a green check — the loop is what's highlighted, not the vault -->
+![The drill loop: data out of the vault is the proof, not data in](/diagrams/restore-or-rumour.svg?v=1)
+
+## What the drill actually catches
+
+The failures a drill surfaces are precisely the ones the nightly tick can't see. A repository still locked
+by a dead process from last week's crashed run — every subsequent "successful" backup queued behind a stale
+lock. A prune step that had been silently failing. And the one that matters most on the worst day: whether
+the *key* is where the recovery plan says it is. Every one of those was invisible from the green-tick view,
+and every one is a nothing-burger to fix on a Tuesday afternoon — and an incident report if discovered
+during a real recovery.
+
+There's a quieter design lesson the drill enforces too: back up **state, not just volumes**. A database dump
+you can restore beats a filesystem snapshot you have to forensically reassemble; the drill makes you notice
+the difference, because the drill is where you actually try.
+
+## One key to rule the recovery
+
+Consolidation concentrates risk on purpose — one NAS, one repository format, one passphrase unlocking every
+cluster's history. That's a fine trade *if* the key management is treated with the same rigour as the data:
+the passphrase lives offline, not in any repository it unlocks, and the recovery plan starts from "a bare
+machine and the key", assuming nothing else survived. A backup encrypted with a key stored next to it is a
+padlock with the key taped to the shackle.
+
+## The principle
+
+Trust receipts, not reports. Any system whose entire purpose is a future emergency — backups, break-glass
+accounts, failovers, restore paths — is untested by definition on every normal day, so you have to
+manufacture the test: drill it, on a schedule, with a real pass/fail, and treat a failed drill as a real
+incident. The nightly green tick is the rumour. The drill is the receipt.
+
+*Live across the homelab: per-cluster restic repositories on the NAS over SMB, nightly CronJobs, restore
+drills run before each cutover — thousands of files pulled back and verified per cluster, a stale-lock
+failure caught and self-healing added, the passphrase held offline.*