00Cold open — the build that "hung"
Nobody sets out to run a registry. You inherit the need the day a CI build stalls for no reason you can see.
The lab's website builds from an ordinary Dockerfile. One base image came from
Docker Hub, pulled anonymously, and one evening the build simply... sat there. No
error visible. No progress. Here is the accurate version: the registry does not go
silent when it rate-limits you — it answers loudly, with an explicit 429 and the
string toomanyrequests. The silence
was manufactured closer to home, by a retry layer swallowing that answer and trying
again, backoff stacked on backoff, until a routine build read like a hung machine.
The swallower, recovered from the build script's own history, was a double retry
stack: an outer shell wrapper allowing four attempts with a growing backoff, wrapped
around the build tool's own three retries — up to twelve tries before anything
surfaced. It was added, in fairness, to ride out transient 502s; nets that catch 502s
catch 429s too, and a limit is not transient.
retry() {
local n=0 max=4
until "$@"; do
n=$((n+1)); [ "$n" -ge "$max" ] && { echo ">> failed after ${max} attempts" >&2; return 1; }
echo ">> attempt ${n} failed, retrying in $((n*8))s..." >&2; sleep $((n*8))
done
}
retry buildah build --retry 3 --retry-delay 5s ...
# four attempts, each containing three - the 429 answered every one of them
The cause was boring and invisible: anonymous pulls from the world's most popular registry are rate-limited, and the base image tag was not pinned, so every single build asked Docker Hub the same question again. The fix for THAT bug is nearly one line — pin the digest. Nearly, because pinning only ends the conversation when a local cache already holds the answer; an ephemeral runner with a cold store still asks Hub for the manifest behind that digest every time. The pin stops the re-resolution, not the round-trip. Which surfaced the sharper question:
The answer a production platform gives is a mirror: one machine that fetches an image once, caches it, and serves every later request from the shelf. The lab built one. This dossier takes it apart.
The mirror did not fix the red X in CI. That red turned out to be a separate upstream bug in the CI system's log finalise step — cosmetic, tolerated, documented. Two problems, one symptom. Diagnose them separately or you will "fix" the wrong one and declare victory.
01Anatomy — one cache, five upstreams
The mirror here is zot: a single OCI registry, one namespace on the platform, one 50-gigabyte cache volume, digest-pinned like everything else in the fleet. It fronts five public registries — Docker Hub, GitHub's, Quay, the Kubernetes project registry, and NVIDIA's — and it works in one of three modes per upstream:
- onDemand — pure pull-through. A miss fetches from upstream, caches, serves. The second pull is instant and never leaves the building.
- polled sync — cache plus refresh on an interval, bounded by tag filters so a poll cannot drag a whole vendor catalogue into the shelf.
- pre-seed — sync-only, for images that must be present before anything asks.
Docker Hub gets onDemand ONLY. It has no catalogue API worth polling and the rate limits that started this story punish enthusiasm.
The storage settings are not defaults; each one bought something:
"storage": {
"commit": true,
"dedupe": true,
"gc": true,
"gcDelay": "1h",
"gcInterval": "24h"
}
commit forces writes to disk before
acknowledging — crash-safety for a box that might lose power.
dedupe hard-links identical layers, which
matters enormously when five upstreams ship the same base layers under different
names. Garbage collection is delayed so a slow client mid-pull never watches its
blobs vanish.
The consumers point at the mirror in their runtime config. On this platform that is a machine-level mirror list — and the FALLBACK SEMANTICS hiding behind this short stanza are the most load-bearing thing in the whole dossier. Hold that thought for chapter 02.
The decision this explains: the fallback is the platform's implicit default — the design choice is refusing to disable it.
machine:
registries:
mirrors:
docker.io:
endpoints:
- https://zot.bztmon.org
ghcr.io:
endpoints:
- https://zot.bztmon.org
Read that carefully: the mirror is the ONLY listed endpoint, and that is still not
a hard dependency — because this platform falls back to the origin registry
IMPLICITLY unless you set skipFallback: true.
The lab's first version of this config listed the upstream as an explicit second
endpoint; the same-day refinement removed it after realising it merely rendered a
duplicate of what the default already guaranteed. The lesson generalises: know which
of your safety nets are things you built, and which are defaults you merely have not
broken.
Upstream credentials — a Docker Hub login to lift rate limits, a vendor key for the GPU registry — live in ONE mounted secret, delivered by the cluster's secret operator under logical names, never in the runtime config. That split is not tidiness; it follows a reload asymmetry worth stating precisely. On this platform, the mirror ENDPOINT list hot-reloads — apply it and the runtime picks it up live, no reboot. Node-level registry AUTH does not: credentials in the machine config take effect only after a reboot. The mounted secret rotates with a restart of one pod. So endpoints and pod-side secrets can move cheaply; node auth is the expensive one — and that asymmetry, not neatness, decides where each credential lives and the whole rollout order in chapter 02. Both behaviours were observed on Talos v1.13.4 with containerd 2.2.4 (June–July 2026); treat the exact split as version-dependent and re-test on upgrades.
In production this is not one pod with a volume. The mirror sits on dedicated storage — a NAS-class array or object store — sized for the whole estate's catalogue, and it is itself backed up, because in a disconnected site the mirror IS the software supply. The lab's 50Gi local volume with a nightly backup is the same organ, one size smaller.
02Chicken and egg — four loops, four answers
Every infrastructure service eventually meets the question: what do you depend on, and what happens when you ARE the dependency? The mirror has four of these loops, and each one got a different answer. This chapter is the reason this page exists.
Loop 1 — the mirror's own image comes through the mirror
zot runs as a container. That container's image lives on one of the very registries zot mirrors. So when the node that hosts zot boots, it asks... zot. Which is not running, because the node is booting.
The answer is the fallback you met in chapter 01: mirror listed, origin
implicit. The runtime tries the mirror, fails fast, and falls through to the real
registry because nobody set skipFallback: true.
The loop breaks itself, and the thing that breaks it is a platform default doing
quiet duty as a bootstrap protocol — the design decision here is the restraint of
not turning it off.
This was proven by accident. A config change shipped with a flag the registry refused to start with (
preserveDigest demands
http.compat — a real pairing rule,
learn it from this page instead of from the crash). The mirror crashlooped. And
nothing else broke: every node quietly fell through to upstream and the fleet never
noticed. An unplanned failover test, passed. The postmortem produced the standing
rule: run zot verify on the config BEFORE
merging, in a throwaway pod, every time.Loop 2 — the recovery tooling deliberately ignores the mirror
The fleet's rescue tooling — the automation that health-checks and rebuilds the platform — could pull its execution image through the mirror like everything else. It does not. Its image is cached on the operations box, outside the cluster entirely.
Because a recovery tool that depends on the thing it recovers is not a recovery tool. It is a passenger.
Loop 3 — you cannot turn authentication on everywhere at once
The target state is a mirror that refuses anonymous pulls. The path there is a sequencing puzzle:
- Auth in the node's registry config requires a REBOOT to take effect — mirror endpoints reload live, credentials do not. (Asymmetries like this decide rollout order; find them before you start.)
- The runtime's fall-back-on-401 behaviour is unreliable enough to carry open upstream issues — so a node whose auth is wrong does not gracefully degrade, it just fails to pull.
- Therefore: anonymous read STAYS ON while every node gets its auth config and its reboot, and each node must PROVE itself before it counts. And here the gate hides a trap of its own: while anonymous read is still on, pulling an ordinary repo proves NOTHING about auth — a node whose credentials never applied sends no authorisation header at all, the mirror happily serves it as an anonymous reader, and the gate false-passes the exact node it exists to catch. The canary must therefore live in a repo whose access policy DENIES anonymous read, so a successful pull can only mean an authenticated pull. Pass THAT, and the node is trusted. Only when every node has passed does anonymous flip off fleet-wide.
The lab's live policy today is the pre-flip shape — one glob, anonymous read on — and the canary repo gets its own entry carrying NO anonymous policy, so its pulls demand credentials while the rest of the shelf stays open:
"accessControl": {
"repositories": {
"**": {
"anonymousPolicy": ["read"],
"policies": [
{ "users": ["zot-push"], "actions": ["read", "create", "update"] },
{ "users": ["zot-pull"], "actions": ["read"] }
]
},
"canary/**": {
"policies": [
{ "users": ["zot-pull", "zot-push"], "actions": ["read"] }
]
}
}
}
That per-node proof is a hard gate precisely because the failure mode is silent. A node that looks fine and cannot pull is a time bomb with a pleasant dashboard.
The decision this explains: the gate is per-node, never fleet-wide — one unproven node under a fleet-wide flip is an outage wearing a green tick.
node A (auth applied, rebooted): $ crictl pull zot.bztmon.org/canary/prompt-forge:0726R1 Image is up to date for sha256:... <- authenticated pull; node A is trusted node B (auth staged in config, no reboot yet): $ crictl pull zot.bztmon.org/canary/prompt-forge:0726R1 E... failed to pull ...: 401 Unauthorized <- node B reboots before it counts
Loop 4 — the mirror's host is also the mirror's customer
The node that hosts the mirror boots its own workloads through it — including, in this lab, the tunnel that serves the public website. So "restart the mirror's node" carries a blast radius far beyond the mirror. The runbook for that reboot lists every public-facing thing that rides on it, and the order they come back.
Draw the dependency arrows for your own estate. The ones that surprise you are the ones that will page you. Sequencing, here, is a first-class engineering artefact.
Production separates these concerns physically: the mirror on its own storage appliance, per-site edge nodes pulling FROM it, no workload sharing its host. The lab collapses them onto one node and manages the consequence with sequencing and runbooks. Same physics, different budget.
03The traps — each rule has a scar
"The registry is up" is not "the mirror works." The worst incident in this mirror's life: an in-flight upstream sync wedged, and from then on EVERY pull of ANY image from that upstream hung — while the registry's health endpoint returned 200 the entire time. The log line, when found, was almost poetic:
"image already demanded, waiting on channel" $ kubectl -n zot rollout restart deploy zot <- the fix, instantly effective # probe an actual manifest, not the health endpoint: $ curl -sI https://zot.bztmon.org/v2/library/busybox/manifests/latest HTTP/2 200 <- THIS is "the mirror works"
Monitor the thing you actually need — a manifest fetch — not the process's opinion of itself.
Digests can change in transit. A mirror that converts image formats on sync
breaks every digest pin and signature downstream of it — silently. The pairing
that prevents it (preserveDigest +
http.compat) is exactly the pairing from
the chapter-02 crashloop. One decision, two scars.
The cache grows until the disk ends. onDemand means every image anyone ever pulls joins the shelf, and this registry does not evict least-recently-used blobs on its own — garbage collection reclaims orphaned data, not old-but-valid images. The receipt from the lab's live config: the storage section carries gc settings and NO retention section at all. That is a real answer — the current policy is "keep everything", chosen implicitly, and the 50-gigabyte volume has never yet been pushed to its high-water mark, so what happens there remains genuinely unobserved. A volume size is a budget, not a policy, and the two retention behaviours on this page — this one and the keep-rule inversion below — are ONE open policy decision: say explicitly what is kept, what is deletable, and what happens when the disk fills, before the disk answers for you.
Retention flips its meaning on the first rule. No retention policy: keep everything. ONE keep-rule on one repo: everything not matching is now DELETABLE. Adding a policy is not narrowing — it is inverting. Write explicit keep coverage per repo pattern, or write none — and read this trap together with the cache-growth one above; they are the two halves of the same unanswered question.
Dedupe is not a toggle. Flipping it on existing storage triggers a full relink pass under lock — pushes can hang half an hour behind it. Maintenance window, or leave it alone.
A tag and a digest in the same reference used to miss the cache — and this trap carries its own moral about traps. The historical mechanism: the runtime routes to mirrors by HOST, so the combined reference reached the mirror, which rejected it as unresolvable ("repository name not known", upstream issue zot 2584), and the runtime fell through to the origin — the fallback that saves chapter 02 quietly defeating the cache. Re-tested against this lab's current registry on 2026-08-24: the combined reference now pulls clean; the failure did not reproduce on v2.1.17. The rule survives in weaker form — prefer one reference form, know your version — but the stronger lesson is that a trap list is perishable. Re-run your own scars occasionally; some of them have healed.
Credentials drift when the hash and the plaintext live apart. The push credential is stored hashed; the drift surfaced on the first of July — plaintext lost, hash still standing — and with no way to prove or re-mint the pair, pushes stayed dead until the rebuilt rotation flow landed on the thirteenth. Twelve days. The receipt that it worked: three separate applications shipped to the fleet the SAME day the credential came back. Now hash and plaintext live side by side in the secret store, rotated together, by a human — the sync identity is deliberately read-only and CANNOT write secrets, so no automation can half-rotate them again.
The cache is invisible in the UI. The web interface shows pushed
repositories only; on-demand cached images do not appear. Cosmetic — but the
first time you look, you will conclude the mirror is empty. Verify with the API
(/v2/<name>/tags/list), not the browser.
04The prod translation — same organ, grown up
Everything above is one lab, one node, one cache. Scale the pattern honestly and it becomes the supply chain of every disconnected or edge estate:
- The mirror lives on real storage — an appliance with parity and snapshots, because at a disconnected site the mirror is not an optimisation, it is the only source of software that exists.
- Each remote site runs its OWN small node, pulling from the central mirror on a schedule, so a site survives its WAN link dying with its full catalogue local.
- Authentication is day one, not phase two — the lab's staged flip is what retrofitting looks like; greenfield does not get that luxury or that risk.
- Promotion is by digest, not by tag. Production estates rewrite image references to pinned digests and promote by copying digests between registries — a tag is a suggestion, a digest is a fact. The lab's digest-pinning discipline is the same muscle at one-tenth the ceremony.
The lab's prettiest trick — mirror listed, origin implicit, the loop that breaks itself — assumes an upstream EXISTS to fall through to. At a genuinely disconnected site there is no second endpoint; if the mirror's own image is not already on the host, nothing will ever fetch it. Production answers this the unglamorous way: the mirror's image is pre-seeded into the host's local image store at provisioning time (or run as a static workload imported straight from disk), or Loop 2's recovery-box pattern is applied to the mirror itself — its image kept OFF the platform it serves. Loop 1's implicit fallback is a connected-world luxury; the airgap makes you choose Loop 2's discipline whether you like it or not.
The decision this explains: per-site nodes exist so a cut link idles NOTHING — pulls never cross the WAN at pull time at all.
The lab is not a toy version of production. It is production with one of everything. The loops in chapter 02 exist at every scale; the only thing that changes is how expensive they are to ignore.
05Close — the canary
The mirror's proof-of-life is not a dashboard. When a node powers on, the fleet's gate makes it pull one known, first-party, mirror-only image through its own runtime — the same path a real workload would take, auth and all. Pass, and the node is a member of the fleet. Fail, and it is a machine that happens to be switched on.
That is the whole dossier in one sentence. A platform is a chain of machines feeding each other bytes, and the mirror is the pantry. Stock it deliberately, prove it constantly, and never let the recovery crew depend on it for lunch.