---
title: "A backup you haven't restored is a rumour"
date: 2026-07-03
summary: "Backup jobs report success every night for years — and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught."
tags: ["backups", "disaster-recovery", "kubernetes", "restic", "homelab"]
hero: "/blog/restore-or-rumour.webp?v=1"
heroAlt: "A dim vault with a small tray of retrieved data-crystals under an examination lamp — the proof is what was brought back OUT, not what sits sealed inside."
draft: false
---


Ask an estate whether it has backups and you'll get a confident yes: the job runs nightly, the dashboard is
green, the repository is growing. Ask when anyone last *restored* one and the room goes quiet.

Here's the uncomfortable framing: an unverified backup is not evidence — it's a rumour. Someone once said
the data was safe, the claim got repeated nightly by a cron job, and everyone chose to believe it. The green
tick proves a process exited zero. It says nothing about whether the bytes at rest can be turned back into a
running application on the worst day of the year.

## The restore drill is the acceptance test

When I consolidated the homelab's backups onto the NAS — every cluster's application state streaming into
one repository per cluster — the job wasn't "done" when the first snapshot landed. It was done when a
scratch pod pulled thousands of files back *out* of the repository, the configs checked intact, and the
services proved recoverable from nothing but the backup and its key. Both clusters. Before the old backup
path was retired, not after.

That's the rule worth writing down: **a backup pipeline ships with its restore drill, the same way code
ships with its tests.** Not a runbook that describes a restore — an actual rehearsal, on the real
repository, with a pass/fail outcome.

<!-- DIAGRAM: nightly snapshot arrows into a repository; a drill loop pulling a snapshot back OUT into a scratch pod with a green check — the loop is what's highlighted, not the vault -->
![The drill loop: data out of the vault is the proof, not data in](/diagrams/restore-or-rumour.svg?v=1)

## What the drill actually catches

The failures a drill surfaces are precisely the ones the nightly tick can't see. A repository still locked
by a dead process from last week's crashed run — every subsequent "successful" backup queued behind a stale
lock. A prune step that had been silently failing. And the one that matters most on the worst day: whether
the *key* is where the recovery plan says it is. Every one of those was invisible from the green-tick view,
and every one is a nothing-burger to fix on a Tuesday afternoon — and an incident report if discovered
during a real recovery.

There's a quieter design lesson the drill enforces too: back up **state, not just volumes**. A database dump
you can restore beats a filesystem snapshot you have to forensically reassemble; the drill makes you notice
the difference, because the drill is where you actually try.

## One key to rule the recovery

Consolidation concentrates risk on purpose — one NAS, one repository format, one passphrase unlocking every
cluster's history. That's a fine trade *if* the key management is treated with the same rigour as the data:
the passphrase lives offline, not in any repository it unlocks, and the recovery plan starts from "a bare
machine and the key", assuming nothing else survived. A backup encrypted with a key stored next to it is a
padlock with the key taped to the shackle.

## The principle

Trust receipts, not reports. Any system whose entire purpose is a future emergency — backups, break-glass
accounts, failovers, restore paths — is untested by definition on every normal day, so you have to
manufacture the test: drill it, on a schedule, with a real pass/fail, and treat a failed drill as a real
incident. The nightly green tick is the rumour. The drill is the receipt.

*Live across the homelab: per-cluster restic repositories on the NAS over SMB, nightly CronJobs, restore
drills run before each cutover — thousands of files pulled back and verified per cluster, a stale-lock
failure caught and self-healing added, the passphrase held offline.*
