The Mirror

01The build that "hung"

The mirror as a machine: upstream intakes, one cache shelf, one serving nozzle.

Nobody sets out to run a registry. You inherit the need the day a CI build stalls for no reason you can see.

This site's own build pulls one base image from Docker Hub, anonymously. One evening the build sat there - no visible error, no progress. The accurate version of that story: Docker Hub does not go silent when it limits you. It answers with an explicit HTTP 429 and the error code toomanyrequests; under the current published limits, anonymous pulls get 100 manifest requests per six hours per IPv4 address, and the accounting is per manifest GET. The silence was manufactured on my side, by retries swallowing that answer.

The swallower, recovered from the build script's git history, was a double retry stack: an outer shell wrapper allowing four attempts, wrapped around buildah build --retry 3. Buildah's flag counts retries, not attempts - one initial try plus up to three retries, and it applies to registry push/pull operations. Four outer attempts, each containing up to four inner attempts: up to sixteen requests for one consistently failing pull, each one billed against the same rate limit it was trying to outlast. Retry amplification turns a throttle into an outage and hides the evidence.

retry() {
  local n=0 max=4
  until "$@"; do
    n=$((n+1)); [ "$n" -ge "$max" ] && { echo ">> failed after ${max} attempts" >&2; return 1; }
    echo ">> attempt ${n} failed, retrying in $((n*8))s..." >&2; sleep $((n*8))
  done
}
retry buildah build --retry 3 --retry-delay 5s ...
# --retry 3 = one attempt + three retries; the wrapper multiplies that by four.
# A 429 answered every one of them, and the backoffs read as a hang.

The immediate fix was to pin the base image by digest - which stops tag re-resolution, but does not remove the network round-trip: an ephemeral runner with a cold local store still fetches the manifest behind that digest every build. The durable answer is structural:

A production platform answers with a mirror: one registry that fetches once and serves the estate. The lab built one. This page takes it apart - including the parts that turned out not to work the way I first believed.

02What the mirror actually does

The mirror is zot: a single OCI registry in its own namespace, one 50Gi cache volume, running the same digest-pinned deployment discipline as everything it serves. It fronts five public registries in on-demand pull-through mode: a miss fetches from upstream and caches; a hit serves from the shelf. zot also supports polled mirroring (periodic full sync of matching content) and pre-seeding; Docker Hub is on-demand only - upstream documentation is explicit that polled mirroring should not be pointed at Hub.

The storage block, from the live config:

"storage": {
  "rootDirectory": "/var/lib/registry",
  "commit": true,
  "dedupe": true,
  "gc": true,
  "gcDelay": "1h",
  "gcInterval": "24h"
}

commit fsyncs writes before acknowledging - crash safety. dedupe hard-links identical blobs, which pays for itself when five upstreams ship the same base layers under different names. And gc with its two timers looks innocuous here; section 06 is about why it is the most consequential block on this page.

Consumers reach the mirror through the platform's machine-level registry config:

machine:
  registries:
    mirrors:
      docker.io:
        endpoints:
          - https://zot.bztmon.org
      ghcr.io:
        endpoints:
          - https://zot.bztmon.org

The mirror is the only listed endpoint, and that is still not a hard dependency, because Talos documents an implicit final fallback: endpoints are tried in order, "and by default the last implicit endpoint is the original upstream registry", unless skipFallback: true says otherwise. The lab's first version listed the upstream as an explicit second endpoint; the refinement that removed it came from reading the behaviour properly - the default already guaranteed it. Know which of your safety nets you built, and which are defaults you merely have not broken.

One asymmetry decides the whole rollout order in section 07: registry auth in the machine config is documented by Talos as requiring a reboot before the CRI picks it up. Mirror endpoint changes applied live in this lab (observed on Talos v1.13.4 / containerd 2.2.4; the docs are silent on this half, so treat it as an observation, not a guarantee). Credentials the registry consumes from a mounted secret rotate with a pod restart. Put each credential on the side that can move.

03Request routing and five upstreams

Five registries feed one endpoint, which raises a question the first version of this page skated past: when a node asks the mirror for pause:3.10, how does the mirror know whether that means Docker Hub, GHCR, Quay, registry.k8s.io or NGC?

Three facts, all verified:

1. containerd tells the mirror where the request came from - a mirror request carries the original registry as a query parameter: /v2/pause/manifests/3.10?ns=registry.k8s.io. That is documented containerd behaviour.
2. zot v2.1.17 ignores it. There is no handling of the ns parameter in the deployed version's request path, and using it for upstream selection is an open upstream feature request (zot issue 4187) with, at last check, no maintainer response. The URL path alone selects the local repository.
3. This deployment has no per-upstream prefixes. The live sync config gives all five upstreams prefix: "**" and no destination - one flat namespace. On a miss, zot tries the configured upstreams in order until one has the path. Upstream documentation's own multi-registry example instead gives each upstream a distinct destination so the requested path selects the origin.

Why has the flat namespace not bitten? Because the five origins use largely disjoint path conventions - Hub's official images live under library/ (the implicit prefix behind bare names like alpine), NGC content sits under nvidia/, registry.k8s.io has its own layout. "Largely disjoint" is a probability, not a guarantee: an organisation name that exists on both GHCR and Quay would collide silently, and the winner would be config order. That is an accepted risk in a lab; it is not a design.

04Bootstrap, fallback and failure modes

The dependency ring: one of the orbiting parts is the mirror itself - its own image is served through the thing it is.

Every infrastructure service eventually meets the question: what do you depend on, and what happens when you ARE the dependency? The mirror has four such loops, and each got a different answer.

Loop 1 - the mirror's own image comes through the mirror

zot runs as a container whose image lives on a registry zot mirrors. When the hosting node boots, it asks the mirror - which is not running, because the node is booting. The loop breaks on the documented fallback: mirror endpoints exhaust, containerd falls through to the origin. The design decision is restraint - not setting skipFallback: true.

Loop 2 - the recovery tooling deliberately ignores the mirror

The fleet's rescue tooling could pull its execution image through the mirror like everything else. It does not: its image is cached on the operations host, outside the cluster. A recovery tool that depends on the thing it recovers is not a recovery tool.

Loop 3 - authentication cannot flip everywhere at once

Covered properly in section 07; the loop shape: enforcing auth requires every node to carry credentials that only apply after a reboot, so anonymous read must survive until the last node is proven, and the proof itself must be designed not to lie.

Loop 4 - the mirror's host is also the mirror's customer

The node hosting the mirror boots its own workloads through it - including the tunnel that serves this very page. "Restart the mirror's node" therefore carries a blast radius far beyond the mirror, and the runbook for that reboot lists every public-facing thing riding on it and the order they return. Draw the dependency arrows for your own estate; the ones that surprise you are the ones that will page you.

Fallback is a policy decision, not a universal win. This lab keeps it on: better bootstrap survivability, availability through mirror outages, and the cost is real - pulls can silently bypass the mirror (losing cache benefit and any future policy point) and land on origin rate limits. A regulated or disconnected estate makes the opposite call: skipFallback: true, egress restrictions, preseeded release stock, and a tested recovery path, because silent bypass is the failure mode there, not the safety net.

And "fails fast" is not one behaviour. The failure modes differ in symptom, path and consequence:

05Digests, manifests and the supply chain

Precision matters here, because half the traps on this page come from conflating these objects:

• A tag is a mutable, human-readable pointer. Zero, one or many tags can reference one manifest.
• A manifest digest identifies exact manifest bytes. A multi-platform index has its own digest; each platform manifest has another; layers and config blobs have their own again.
• name:tag@digest is client-side grammar, not wire protocol: the OCI distribution spec takes a tag or a digest in the URL. Clients resolve the digest and ignore the tag (Kubernetes documents exactly this), so digest-pinned pulls are immune to tag moves - a fact this page once mislearned as a cache trap and later retested: the historical zot rejection of combined references did not reproduce on v2.1.17. Traps are perishable; retest your scars.
• Digest pinning provides content integrity - not provenance, not authorisation, not availability, and not freedom from network fetches on a cold store.

For a mirror, digests carry one sharp operational rule, learned here as a crashloop: zot converts Docker-schema manifests to OCI by default, and conversion changes the digest - silently breaking every digest pin and signature downstream. The pairing that prevents it (preserveDigest: true per upstream, with http.compat: ["docker2s2"]) is mandatory in both directions: the registry refuses to start with one and not the other. This deployment runs both, on all five upstreams - observed in the live config.

Preserving digests does not mean the supply chain travels whole. OCI 1.1 referrers - signatures, SBOMs, attestations attached via the subject field and served by the referrers API - are separate objects with their own discovery path. A pull-through cache that mirrors manifests and blobs does not automatically carry them. Nothing here verifies signatures today; if it did, disconnected verification would also need the trust material carried locally (a public key, or a trusted-root bundle for a private signing stack). That is stated as a boundary, not an aspiration.

06Retention, GC and the digest-only trap

Cached bricks queued behind a jammed intake: availability problems here are quiet, not loud.

This section corrects the largest error in this page's own first edition. I wrote: "no retention config means keep everything". That is true for tags, and dangerously false for everything else - and "everything else" includes precisely the content this mirror exists to hold.

The verified semantics, from upstream documentation and the deployed version's source:

• With no retention configuration, all tags are retained - and all untagged manifests are deleted by garbage collection (unless referenced by an index or artifact) once they are older than gcDelay. This deployment's delay is the 1-hour default, with GC sweeping daily.
• Digest-only pull-through entries are stored as untagged manifests. That is the exact wording of the upstream fix's problem statement: pull an image through the mirror by digest, and the cache entry has no tag to protect it.

Put the pieces together with section 04's revalidation fact and the deployed behaviour is this: tag pulls are cached but always phone upstream anyway; digest pulls are served locally but their cache entries are eligible for eviction within hours. On v2.1.17, this mirror is a rate-limit shield and a latency win - it is not yet a disconnection shelf, and the first edition of this page was wrong to imply otherwise.

Retention policy semantics, stated precisely because two nearby systems use opposite rules:

• Retention policies match per repository, first match wins - order the list most-specific-first. A repository matching no policy keeps its tags.
• Within a matched repository, one keepTags rule inverts the default: non-matching tags in that repository become deletable. The inversion is scoped to the repository, not global - the first edition of this page overstated it.
• Untagged manifests are governed separately (deleteUntagged, default true), and on the deployed version no retention rule can protect them. Pull-aware keepUntagged exists upstream from v2.1.19.
• Authorisation policies (section 07) use longest match, not first match. Two adjacent config blocks, two opposite precedence rules. Label which one you are reasoning about.

Mitigations, honestly ranked: upgrade to v2.1.19+ and configure keepUntagged with pull-activity rules (the designed fix); until then, widen gcDelay/retention.delay so eviction pressure drops (the upstream maintainer's own interim suggestion), or set deleteUntagged: false and accept that the cache only grows - a capacity trade, not a free lunch. Disabling GC entirely trades eviction for guaranteed storage exhaustion with no reclaim path; it is listed here to be argued against.

07The authentication migration

The proving gate: a node counts as migrated when an authenticated pull succeeds where an anonymous one cannot.

Target state: the mirror refuses anonymous pulls. Current state: anonymous read on, with per-node credentials staged in machine configs (inert until each node's reboot - the documented behaviour). The migration is a sequencing problem with one trap at its centre:

While anonymous read is on, an ordinary pull proves nothing about auth. A node whose credentials never applied sends no authorisation header, gets served as an anonymous reader, and false-passes the gate. The canary must live in a repository that denies anonymous read, so a successful pull can only mean an authenticated pull.

That design is now upstream-validated rather than assumed: zot authorisation resolves per-repository policies by longest match - the most specific path pattern wins, and ** is explicitly the default policy for anything unmatched. A canary/** entry that grants named identities read and carries no anonymousPolicy therefore denies anonymous on that path while the glob keeps the rest of the shelf open. Maintainer guidance confirms anonymous and authenticated access are evaluated independently. The staged policy:

"accessControl": {
  "repositories": {
    "**": {
      "anonymousPolicy": ["read"],
      "policies": [
        { "users": ["zot-push"], "actions": ["read", "create", "update"] },
        { "users": ["zot-pull"], "actions": ["read"] }
      ]
    },
    "canary/**": {
      "policies": [
        { "users": ["zot-pull", "zot-push"], "actions": ["read"] }
      ]
    }
  }
}

The gate, per node, and what each step proves:

1. Stage credentials in the node's machine config (inert; documented as requiring reboot).
2. Reboot the node at a planned window.
3. Prove identity: pull the protected canary through the node's own runtime. Success = authenticated (anonymous cannot); failure = 401, back to step 1. The expected split, to be confirmed against observed responses when the test runs: 401 for missing/wrong credentials, 403 for a valid identity lacking the action - upstream docs state the 403 case explicitly only for OIDC identities, so the basic-auth 403 boundary is listed in the open-verification register rather than asserted.
4. Prove interception separately: the canary pull is a direct reference to the mirror, so it cannot prove that docker.io/... references are being routed through the mirror at all. That second proof needs an original-upstream reference pulled on the node plus the mirror's request log showing it arrive - and a cold runtime store, because node-local content will satisfy the pull without any network and fake a pass.
5. Only when every node passes both proofs does anonymousPolicy come off the glob. Rollback is the previous config commit, and it is named before the flip, not after.

Behind the gate sit five separate trust domains, deliberately not shared: node pull credentials (machine config, reboot-bound), CI push credentials (human-held login), the mirror's own upstream credentials (one mounted secret, pod-restart-bound), TLS trust (estate wildcard, standard roots), and human admin access (SSO in front of the UI). The push credential carries this page's oldest scar: its hash and plaintext once drifted apart, rotation became impossible, and pushes stayed dead for twelve days until the rebuilt flow landed - three applications shipped the same day it came back. Hash and plaintext now live side by side, rotated together by a human, and the automation identity that syncs secrets is read-only by design so it can never half-rotate them again.

08Observability that means something

"The registry is up" is the least useful sentence in this page. The worst incident in this mirror's life: an in-flight sync wedged, every pull from that upstream hung, and the health endpoint returned 200 throughout. The log line was almost poetic:

"image already demanded, waiting on channel"
$ kubectl -n zot rollout restart deploy zot   <- recovery action (not a root cause)
# probe an actual manifest, not the process's opinion of itself:
$ curl -sfS --max-time 10 -o /dev/null -w '%{http_code}\n' \
    https://zot.bztmon.org/v2/library/busybox/manifests/latest \
    -H 'Accept: application/vnd.oci.image.index.v1+json'
200                                           <- THIS is "the mirror works"

Honesty about that incident, upgraded by reading the source: the log line is coalescing by design - concurrent requests for one image join the first sync rather than duplicating it, and since the deployed version syncs run on a detached background context with a three-hour default timeout, surviving client disconnects. Waiters blocking on a genuinely stalled sync until that timeout is consistent with what we saw; the restart was a recovery action, and the root cause was never isolated. Two knobs this config does not currently set - sync maxRetries (disabled by default upstream) and a tighter syncTimeout - are the first things a recurrence should change.

What this deployment actually exposes, verified against the running version's source: /livez, /readyz and /startupz exist as real health endpoints (absent from the docs at this version - a documentation gap, not an invention), alongside the spec's /v2/. The Prometheus metrics extension exists upstream (real series names include zot_http_requests_total, zot_repo_storage_bytes, zot_repo_downloads_total, zot_storage_lock_latency_seconds) - and is not enabled in this deployment. Wiring it, with scrape auth, is on the open list; no dashboard here pretends otherwise.

The layered checks that would actually mean something, each answering one question:

The test matrix for the caching claims - each cell is an experiment, not an assumption (status: the two digest rows are documented upstream and consistent with observed behaviour here; the disconnected rows have not been run in this lab):

09The production translation

A central curated source and site-local registries: each site holds its own shelf, so a cut WAN idles nothing.

The first edition of this section asserted what "production" does. That was the wrong register: production chooses from patterns against requirements. The honest version:

• Single registry, accepted failure domain - exactly this lab's shape. Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or to a paused deploy window (fallback off) and that cost is accepted in writing.
• Sync-based HA - upstream documents active/standby and active/active pairs of independent zot instances, each with its own storage, mirroring each other behind a load balancer; the documented caveat is the synchronisation window between polls, which bounds what a failover can lose.
• Scale-out clustering - upstream's other documented shape: instances shard repositories by hash and proxy to the owner, classically over shared S3-compatible storage. That is horizontal scale for load; the shared storage is then the availability story, which is a different property than the HA pair above. Naming which property you are buying is the design act.
• Edge estates - a central curated source promotes releases by digest; each site runs a small local registry holding its release set, synced on a schedule, so a severed WAN idles nothing at pull time. A disconnection claim is only valid when the required content, its referrers if policy demands them, trust roots, credentials and the registry's own bootstrap image are already local - the checklist from diagram D, applied per site.

Whatever the topology: recovery time and recovery point get numbers before an incident provides them; backups are restore-tested copies off the failure domain (array snapshots and RAID protect against disks, not against the array, the site, or an operator error - they are inputs to a backup strategy, not the strategy); capacity is planned against the retention policy from section 06, because "how big does the cache get" is a policy output, not a guess; and the circular dependencies from section 04 are drawn per site, because every one of them exists at every scale - the only thing that changes is how expensive they are to ignore.

10Verified, open, and where the claims come from

Proof of life is one authenticated byte transfer through the real path - not a status page.

When a node powers on, this fleet's gate makes it pull one known, first-party image through its own runtime before it counts as a member. That gate is the whole page in miniature: prove the path, not the process.

Open verification register

1. Digest-entry eviction reproduction in a disposable instance (section 06 procedure).
2. Canary repo negative test: anonymous 401 on canary/** while anonymous read elsewhere still succeeds; observed 401-vs-403 boundaries recorded.
3. Mirror interception proof: original-reference pull on a cold node correlated with the mirror's request log.
4. Disconnected-serve drill: cached digest pull with upstream blocked, disposable environment first.
5. Upgrade evaluation: v2.1.19/v2.1.20 for keepUntagged, with the retention config written and reviewed before the upgrade, not after.
6. Namespace redesign migration plan (per-origin destinations + overridePath), including cache re-warm cost.

Verified against

• zot documentation (versioned): retention · storage · mirroring · authn/authz · high availability · scale-out
• zot upstream issues/PRs: 4148 digest-only eviction · 4191 keepUntagged (v2.1.19) · 4128 tag revalidation · 4187 containerd ns param · 2928 mixed anonymous auth · 2505 demand coalescing
• Talos: pull-through registry cache (v1.13) · containerd: registry hosts and the ns parameter
• OCI: distribution spec · image/distribution 1.1 (referrers) · signing: cosign verify
• Docker Hub usage and limits · buildah build --retry semantics