copy pass: straighten quotes and arrows, thin the dash asides, drop the app-count caption from the constellation
build-and-deploy / build (push) Failing after 13m18s

This commit is contained in:
2026-08-17 23:12:42 +10:00
parent 78e8740408
commit 3cfd59a464
23 changed files with 257 additions and 271 deletions
+13 -13
View File
@@ -1,10 +1,10 @@
---
title: "A backup you haven't restored is a rumour"
date: 2026-07-03
summary: "Backup jobs report success every night for years and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught."
summary: "Backup jobs report success every night for years, and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught."
tags: ["backups", "disaster-recovery", "kubernetes", "restic", "homelab"]
hero: "/blog/restore-or-rumour.webp?v=1"
heroAlt: "A dim vault with a small tray of retrieved data-crystals under an examination lamp the proof is what was brought back OUT, not what sits sealed inside."
heroAlt: "A dim vault with a small tray of retrieved data-crystals under an examination lamp - the proof is what was brought back OUT, not what sits sealed inside."
draft: false
---
@@ -12,33 +12,33 @@ draft: false
Ask an estate whether it has backups and you'll get a confident yes: the job runs nightly, the dashboard is
green, the repository is growing. Ask when anyone last *restored* one and the room goes quiet.
Here's the uncomfortable framing: an unverified backup is not evidence it's a rumour. Someone once said
Here's the uncomfortable framing: an unverified backup is not evidence - it's a rumour. Someone once said
the data was safe, the claim got repeated nightly by a cron job, and everyone chose to believe it. The green
tick proves a process exited zero. It says nothing about whether the bytes at rest can be turned back into a
running application on the worst day of the year.
## The restore drill is the acceptance test
When I consolidated the homelab's backups onto the NAS every cluster's application state streaming into
one repository per cluster the job wasn't "done" when the first snapshot landed. It was done when a
When I consolidated the homelab's backups onto the NAS - every cluster's application state streaming into
one repository per cluster - the job wasn't "done" when the first snapshot landed. It was done when a
scratch pod pulled thousands of files back *out* of the repository, the configs checked intact, and the
services proved recoverable from nothing but the backup and its key. Both clusters. Before the old backup
path was retired, not after.
That's the rule worth writing down: **a backup pipeline ships with its restore drill, the same way code
ships with its tests.** Not a runbook that describes a restore an actual rehearsal, on the real
ships with its tests.** Not a runbook that describes a restore - an actual rehearsal, on the real
repository, with a pass/fail outcome.
<!-- DIAGRAM: nightly snapshot arrows into a repository; a drill loop pulling a snapshot back OUT into a scratch pod with a green check the loop is what's highlighted, not the vault -->
<!-- DIAGRAM: nightly snapshot arrows into a repository; a drill loop pulling a snapshot back OUT into a scratch pod with a green check - the loop is what's highlighted, not the vault -->
![The drill loop: data out of the vault is the proof, not data in](/diagrams/restore-or-rumour.svg?v=1)
## What the drill actually catches
The failures a drill surfaces are precisely the ones the nightly tick can't see. A repository still locked
by a dead process from last week's crashed run every subsequent "successful" backup queued behind a stale
by a dead process from last week's crashed run - every subsequent "successful" backup queued behind a stale
lock. A prune step that had been silently failing. And the one that matters most on the worst day: whether
the *key* is where the recovery plan says it is. Every one of those was invisible from the green-tick view,
and every one is a nothing-burger to fix on a Tuesday afternoon and an incident report if discovered
and every one is a nothing-burger to fix on a Tuesday afternoon, and an incident report if discovered
during a real recovery.
There's a quieter design lesson the drill enforces too: back up **state, not just volumes**. A database dump
@@ -47,7 +47,7 @@ the difference, because the drill is where you actually try.
## One key to rule the recovery
Consolidation concentrates risk on purpose one NAS, one repository format, one passphrase unlocking every
Consolidation concentrates risk on purpose - one NAS, one repository format, one passphrase unlocking every
cluster's history. That's a fine trade *if* the key management is treated with the same rigour as the data:
the passphrase lives offline, not in any repository it unlocks, and the recovery plan starts from "a bare
machine and the key", assuming nothing else survived. A backup encrypted with a key stored next to it is a
@@ -55,11 +55,11 @@ padlock with the key taped to the shackle.
## The principle
Trust receipts, not reports. Any system whose entire purpose is a future emergency backups, break-glass
accounts, failovers, restore paths is untested by definition on every normal day, so you have to
Trust receipts, not reports. Any system whose entire purpose is a future emergency - backups, break-glass
accounts, failovers, restore paths - is untested by definition on every normal day, so you have to
manufacture the test: drill it, on a schedule, with a real pass/fail, and treat a failed drill as a real
incident. The nightly green tick is the rumour. The drill is the receipt.
*Live across the homelab: per-cluster restic repositories on the NAS over SMB, nightly CronJobs, restore
drills run before each cutover thousands of files pulled back and verified per cluster, a stale-lock
drills run before each cutover - thousands of files pulled back and verified per cluster, a stale-lock
failure caught and self-healing added, the passphrase held offline.*