@@ -157,24 +161,27 @@
The mirror as a machine: upstream intakes, one cache shelf, one serving nozzle.
-Nobody sets out to run a registry. You inherit the need the day a CI build stalls for no
-reason you can see.
+I added zot after a CI pull hit Docker Hub's anonymous rate limit and the retry wrapper
+hid the 429. At the time of the incident, this site's build-stage pull was anonymous.
-This site's own build pulls one base image from Docker Hub, anonymously. One evening the
-build sat there - no visible error, no progress. The accurate version of that story: Docker
-Hub does not go silent when it limits you. It answers with an explicit HTTP 429 and the error
-code toomanyrequests; under the current published limits, anonymous pulls get
-100 manifest requests per six hours per IPv4 address, and the accounting is per manifest
-GET. The silence was manufactured on my side, by retries swallowing
-that answer.
+Docker Hub does not go silent when it limits you. It answers with HTTP 429 and the error
+code toomanyrequests. Under the
+current published limits ,
+unauthenticated users get 100 pulls per six hours per IPv4 address (or IPv6 /64); a
+single-platform image counts as one pull, a multi-architecture image counts once per
+architecture pulled, version checks do not count, and a HEAD request can read
+the rate-limit headers without consuming a pull. A separate abuse limiter covers all
+request types with its own 429 form. The silence in this incident was
+manufactured on my side, by retries swallowing the answer.
-The swallower, recovered from the build script's git history, was a double retry stack: an
-outer shell wrapper allowing four attempts, wrapped around buildah build --retry 3.
-Buildah's flag counts retries , not attempts - one initial try plus up to three
-retries, and it applies to registry push/pull operations. Four outer attempts, each
-containing up to four inner attempts: up to sixteen requests for one
-consistently failing pull, each one billed against the same rate limit it was trying to
-outlast. Retry amplification turns a throttle into an outage and hides the evidence.
+The swallower, recovered from the build script's git history, was a double retry stack:
+an outer shell wrapper allowing four attempts, wrapped around
+buildah build --retry 3. Buildah's flag counts retries - one initial
+try plus up to three more, applying to registry push/pull operations. Four outer attempts,
+each containing up to four inner attempts: up to sixteen requests for one
+repeatedly failing registry operation (a build with several pulls can produce more), each
+billed against the limit it was trying to outlast. Retry amplification turns a throttle
+into an outage and hides the evidence while doing it.
Receipt · the retry stack, as committed to the repo
retry() {
@@ -188,30 +195,21 @@ retry buildah build --retry 3 --retry-delay 5s ...
# --retry 3 = one attempt + three retries; the wrapper multiplies that by four.
# A 429 answered every one of them, and the backoffs read as a hang.
-The immediate fix was to pin the base image by digest - which stops tag re-resolution, but
-does not remove the network round-trip: an ephemeral runner with a cold local store still
-fetches the manifest behind that digest every build. The durable answer is structural:
-
-SAY IT Why does a fleet of machines ask the public
-internet for the same bytes, hundreds of times, forever?
-
-A production platform answers with a mirror: one registry that fetches once and serves the
-estate. The lab built one. This page takes it apart - including the parts that turned out
-not to work the way I first believed.
+Pinning the base image by digest stopped the tag re-resolution but not the round-trip:
+an ephemeral runner with a cold store still fetches the manifest behind that digest every
+build. The repeated origin pulls justified a shared cache. This page takes it apart,
+including the parts that turned out not to work the way I first believed.
FIELD NOTE
+tolerated, documented. Two problems, one symptom; diagnose them separately.
02 What the mirror actually doesThe mirror is zot: a single OCI registry in its own namespace, one 50Gi cache volume,
-running the same digest-pinned deployment discipline as everything it serves. It fronts five
-public registries in on-demand pull-through mode: a miss fetches from
-upstream and caches; a hit serves from the shelf. zot also supports polled mirroring
-(periodic full sync of matching content) and pre-seeding; Docker Hub is on-demand only -
-upstream documentation is explicit that polled mirroring should not be pointed at Hub.
+The mirror is zot: one registry, one 50Gi cache volume, digest-pinned like everything
+it serves. Five public registries front it in on-demand pull-through mode -
+a miss fetches and caches, a hit serves from the shelf. zot also offers polled mirroring
+and pre-seeding; Docker Hub is on-demand only, per its documentation.
The storage block, from the live config:
@@ -224,12 +222,16 @@ upstream documentation is explicit that polled mirroring should not be pointed a
"gcInterval": "24h"
}
-commit fsyncs writes before acknowledging - crash safety. dedupe
-hard-links identical blobs, which pays for itself when five upstreams ship the same base
-layers under different names. And gc with its two timers looks innocuous here;
-section 06 is about why it is the most consequential block on this page.
+commit asks zot to commit writes to disk immediately instead of relying on
+buffered flushing; it narrows the buffered-write window, while end-to-end power-loss
+durability still depends on the filesystem, volume and disk. dedupe uses hard
+links on local filesystem storage (remote backends implement it differently), saving
+capacity when five upstreams ship the same base layers under different names - at the cost
+of a startup reconciliation pass that matters operationally when toggled on existing data.
+And gc with its two timers looks innocuous here; section 06 is about why it is
+the most consequential block on this page.
-Diagram A · system context
+
Diagram A · system context
System context: consumers, the mirror, its storage, five upstreams, and the trust boundaries between them
CI and Kubernetes nodes pull from the zot mirror over the LAN. The mirror stores content on a 50Gi volume and syncs on demand from five upstream registries across the internet boundary. Anonymous read is currently allowed inbound; outbound upstream credentials live in one mounted secret. The origin-fallback path from nodes directly to upstreams is shown dashed.
@@ -261,10 +263,9 @@ section 06 is about why it is the most consequential block on this page.
solid = normal pull path · dashed cyan = on-demand sync on miss · dashed magenta = origin fallback
-
The decision this explains: one cache serves two consumer classes, and the
-two credential domains (nodes-to-mirror, mirror-to-upstreams) never mix.
+
One cache, two consumer classes; the two credential domains never mix.
Consumers reach the mirror through the platform's machine-level registry config:
+Consumers point at it via the platform's machine-level registry config:
machine:
registries:
@@ -276,52 +277,44 @@ two credential domains (nodes-to-mirror, mirror-to-upstreams) never mix.
-The mirror is the only listed endpoint, and that is still not a hard dependency, because
-Talos documents an implicit final fallback: endpoints are tried in order, "and by default
-the last implicit endpoint is the original upstream registry", unless
-skipFallback: true says otherwise. The lab's first version listed the upstream
-as an explicit second endpoint; the refinement that removed it came from reading the
-behaviour properly - the default already guaranteed it. Know which of your safety nets you
-built, and which are defaults you merely have not broken.
+The mirror is the only listed endpoint, yet not a hard dependency: Talos tries
+endpoints in order "and by default the last implicit endpoint is the original upstream
+registry", unless skipFallback: true. An early version listed the upstream
+explicitly as endpoint two; it was removed as redundant - origin fallback here is a Talos
+default, not an endpoint this project maintains.
-One asymmetry decides the whole rollout order in section 07: registry auth in
-the machine config is documented by Talos as requiring a reboot before the CRI picks it up.
-Mirror endpoint changes applied live in this lab (observed on Talos v1.13.4 /
-containerd 2.2.4; the docs are silent on this half, so treat it as an observation, not a
-guarantee). Credentials the registry consumes from a mounted secret rotate with a pod
-restart. Put each credential on the side that can move.
+One asymmetry decides section 07's rollout order. Talos's
+v1alpha1
+machine-config reference states for registry auth: "changes to the registry auth will
+not be picked up by the CRI containerd plugin without a reboot" - matching what this lab
+saw on v1.13.4. Mirror endpoint changes applied live here (same versions; the
+docs are silent on that half, so it stays a lab observation, not a cross-version
+guarantee). Secrets the registry consumes from a mount rotate with a pod restart. Put each
+credential on the side that can move.
-THE HOMELAB CLAUSE
+One pod, one PVC, one node is an accepted failure domain here, recorded as such;
+section 09 covers what changes when the requirements do.
03 Request routing and five upstreamsFive registries feed one endpoint, which raises a question the first version of this page
-skated past: when a node asks the mirror for pause:3.10, how does the mirror
-know whether that means Docker Hub, GHCR, Quay, registry.k8s.io or NGC?
-
-Three facts, all verified:
+Five registries feed one endpoint: when a node asks the mirror for
+pause:3.10, how does it know which origin that means? Three facts:
-containerd tells the mirror where the request came from - a mirror
-request carries the original registry as a query parameter:
-/v2/pause/manifests/3.10?ns=registry.k8s.io. That is documented containerd
-behaviour.zot v2.1.17 ignores it. There is no handling of the ns
-parameter in the deployed version's request path, and using it for upstream selection is an
-open upstream feature request (zot issue 4187) with, at last check, no maintainer response.
+containerd names the origin - mirror requests carry it as a query
+parameter: /v2/pause/manifests/3.10?ns=registry.k8s.io (documented).zot v2.1.17 ignores it - no ns handling exists in the
+deployed request path; using it for upstream selection is open feature request zot 4187.
The URL path alone selects the local repository.This deployment has no per-upstream prefixes. The live sync config
-gives all five upstreams prefix: "**" and no destination - one
-flat namespace. On a miss, zot tries the configured upstreams in order until one has the
-path. Upstream documentation's own multi-registry example instead gives each upstream a
-distinct destination so the requested path selects the origin.This deployment has no per-upstream prefixes - the live sync config
+gives all five upstreams prefix: "**" and no destination. On a
+miss, zot tries upstreams in config order until one has the path; the docs' own
+multi-registry example instead gives each a distinct destination.
-
+
Upstream map - observed configuration
-Origin Local namespace Outbound auth Sync mode Digest preserved On miss Status Origin Local namespace Outbound auth Sync mode Digest preserved On miss Status docker.io flat (no prefix - collision-ambiguous by construction) Hub login (rate-limit lift) onDemand yes upstreams tried in config order; first that resolves the path wins observed ghcr.io none onDemand yes observed quay.io none onDemand yes observed destination so the requested path selects the origin.
nvcr.io $oauthtoken + key onDemand yes observed
Why has the flat namespace not bitten? Because the five origins use largely disjoint path
-conventions - Hub's official images live under library/ (the implicit prefix
-behind bare names like alpine), NGC content sits under nvidia/,
-registry.k8s.io has its own layout. "Largely disjoint" is a probability, not a guarantee: an
-organisation name that exists on both GHCR and Quay would collide silently, and the winner
-would be config order. That is an accepted risk in a lab; it is not a design.
+The flat namespace has not bitten because the origins use largely disjoint paths -
+Hub's official images live under library/ (the implicit prefix behind bare
+names like alpine), NGC under nvidia/. Largely disjoint is a
+probability: an organisation existing on both GHCR and Quay would collide, and config order
+would pick the winner. Accepted risk, recorded; not a design.
-PROPOSED - NOT YET DEPLOYED destination prefix in zot
-(/docker, /ghcr, ...), and point each Talos mirror at the
-prefixed path with overridePath: true (which stops the automatic
-/v2 append so the prefix survives). Then the requested path itself names the
-origin, and collisions become impossible rather than improbable. The migration cost: every
-cached repository changes its local path, so the cache re-warms.
+PROPOSED - NOT YET DEPLOYED destination prefix per upstream
+in zot (/docker, /ghcr, ...), with each Talos mirror pointed at
+the prefixed path via overridePath: true (suppressing the automatic
+/v2 append). The requested path then names the origin and collisions become
+impossible. Migration cost: every cached repository changes local path, so the cache
+re-warms.
-Diagram B · request flow
+
Diagram B · request flow
Request flow from a node through containerd to the mirror, with hit, miss, revalidation and fallback branches
A pull begins at the node's local content store. On local miss, containerd asks the mirror. A cached digest request is served from mirror storage without upstream contact. A tag request is revalidated against the upstream even when cached, on the deployed version. A storage miss triggers on-demand sync from the matching upstream. If the mirror cannot serve, containerd falls back to the origin registry.
@@ -370,8 +361,7 @@ cached repository changes its local path, so the cache re-warms.
green = fully local · amber = the surprising branch on v2.1.17 · magenta = fallback
-
The amber branch is section 04's punchline: a warm cache does not mean the
-network is out of the story.
The amber branch: a warm cache does not take the network out of the story.
04 Bootstrap, fallback and failure modesThe dependency ring: one of the orbiting parts is the mirror itself - its own image is served through the thing it is.
-Every infrastructure service eventually meets the question: what do you depend on, and
-what happens when you ARE the dependency? The mirror has four such loops, and each got a
-different answer.
+Four circular dependencies live in this design; each got a different answer.
Loop 1 - the mirror's own image comes through the mirror
zot runs as a container whose image lives on a registry zot mirrors. When the hosting
node boots, it asks the mirror - which is not running, because the node is booting. The
loop breaks on the documented fallback: mirror endpoints exhaust, containerd falls through
-to the origin. The design decision is restraint - not setting
-skipFallback: true.
+to the origin. Keeping skipFallback unset is the load-bearing choice.
-FIELD NOTE · OBSERVED preserveDigest enabled but without its mandatory
-partner http.compat - a pairing the registry refuses to start without (the
-documentation is explicit; so was the crashloop). The mirror went down; the fleet quietly
-fell through to upstream and nothing user-visible broke. An unplanned failover exercise,
-passed. The standing rule it bought: validate the config with the registry's own
-verify in a throwaway pod before merging, every time.
+FIELD NOTE · LAB, 2026-06 preserveDigest enabled but without its
+mandatory partner http.compat - a pairing the registry refuses to start
+without. The mirror crashlooped; the fleet fell through to upstream and nothing
+user-visible broke. Since then, every config change runs the registry's own
+verify in a throwaway pod before merging.
-Loop 2 - the recovery tooling deliberately ignores the mirror
-The fleet's rescue tooling could pull its execution image through the mirror like
-everything else. It does not: its image is cached on the operations host, outside the
-cluster. A recovery tool that depends on the thing it recovers is not a recovery tool.
+Loop 2 - the recovery tooling ignores the mirror
+The fleet's rescue tooling does not pull through the mirror: its image is cached on the
+operations host, outside the cluster, so it remains available while zot or its cluster is
+down.
Loop 3 - authentication cannot flip everywhere at once
-Covered properly in section 07; the loop shape: enforcing auth requires every node to
-carry credentials that only apply after a reboot, so anonymous read must survive until the
-last node is proven, and the proof itself must be designed not to lie.
+Enforcing auth needs every node to carry credentials that only apply after a reboot, so
+anonymous read must survive until the last node is proven - and the proof must be designed
+not to lie. Section 07 covers it.
-Loop 4 - the mirror's host is also the mirror's customer
-The node hosting the mirror boots its own workloads through it - including the tunnel
-that serves this very page. "Restart the mirror's node" therefore carries a blast radius
-far beyond the mirror, and the runbook for that reboot lists every public-facing thing
-riding on it and the order they return. Draw the dependency arrows for your own estate; the
-ones that surprise you are the ones that will page you.
+Loop 4 - the mirror's host is also its customer
+The node hosting the mirror boots its own workloads through it, including the tunnel
+serving this page - so restarting that node carries a blast radius beyond the mirror, and
+its reboot runbook lists every public-facing dependant and the order they return.
-Diagram D · two bootstrap worlds
+
Diagram D · two bootstrap worlds
Connected fail-open bootstrap versus disconnected preseeded bootstrap, as separate paths
Left: the connected lab path - node boots, mirror miss, implicit fallback to origin, mirror comes up afterwards. Right: the disconnected path - no origin exists; the registry image and release set must be preseeded onto the host or imported from disk before anything else can start.
@@ -438,26 +422,24 @@ ones that surprise you are the ones that will page you.
left = this lab, observed · right = the airgap translation, a design requirement not a deployed claim
-
Loop 1's trick assumes an upstream exists to fall through to. The airgap
-deletes that assumption, and forces Loop 2's discipline onto the mirror itself.
+
Loop 1 assumes an upstream to fall through to; the airgap deletes that
+assumption and forces Loop 2's discipline onto the mirror itself.
Fallback is a policy decision, not a universal win. This lab keeps it on:
-better bootstrap survivability, availability through mirror outages, and the cost is real -
-pulls can silently bypass the mirror (losing cache benefit and any future policy point) and
-land on origin rate limits. A regulated or disconnected estate makes the opposite call:
-skipFallback: true, egress restrictions, preseeded release stock, and a tested
-recovery path, because silent bypass is the failure mode there, not the safety net.
+Fallback is a policy decision. This lab keeps it on for bootstrap
+survivability and availability through mirror outages; the cost is that pulls can bypass
+the mirror unnoticed and land on origin rate limits. A regulated or disconnected estate
+makes the opposite call - skipFallback: true, egress restriction, preseeded
+release stock, a tested recovery path - because there the bypass is the failure mode.
+Failure classes differ in symptom, path and consequence:
-And "fails fast" is not one behaviour. The failure modes differ in symptom, path and
-consequence:
-
-
-Failure User-visible symptom Request path Fallback? Detection Security consequence Mirror pod down none (pulls slower) node -> origin yes, immediate mirror probes red; origin egress rises policy/audit bypass while down
+Failure modes - symptom, path, fallback and consequence
+Failure User-visible symptom Request path Fallback? Detection Security consequence Mirror pod down public pulls may continue transparently; private images, origin limits, DNS or TLS issues can slow or fail them node -> origin after the mirror endpoint fails; elapsed time differs by failure class (refused vs DNS vs TLS vs blackhole) mirror probes red; origin egress rises policy/audit bypass while down Mirror up, sync wedged pulls hang up to sync timeout node -> mirror (blocked) only after timeout manifest probe stalls while /v2/ still answers availability, not integrity Upstream 429 miss/tag pulls fail or crawl mirror -> origin refused fallback hits the same limit zot logs; retry storms amplify none; self-inflicted DoS via retries Upstream down, digest cached none mirror serves locally not needed - none (verified behaviour) Upstream down, tag cached fails/stalls on v2.1.17 mirror insists on revalidating origin also down the test matrix in section 08 availability surprise - the trap of assuming "cached = offline-safe" Upstream 429 miss/tag pulls fail or crawl mirror -> origin refused fallback also reaches the origin and may hit a pull or abuse limit - the quota bucket depends on the node's identity and source IP, not necessarily zot's zot logs; retry storms amplify self-inflicted denial of service via retries Upstream down, digest cached none expected mirror serves locally not needed - none (source-verified short-circuit; disconnected drill still open) Upstream down, tag cached reported on v2.1.17: failure or waiting until timeout, with content possibly served after it mirror revalidates the tag upstream first origin also down the test matrix in section 08 availability surprise for anyone assuming cached means offline-safe (upstream-reported; not yet reproduced here) PVC full new pulls fail, cached OK-ish mirror 5xx on writes yes for missing content capacity metrics (proposed) availability; GC pressure Digest entry GC-evicted silent re-fetch on next pull mirror -> origin re-sync n/a upstream egress for "cached" content rate-limit exposure returns (section 06) Node auth wrong (post-flip) ImagePullBackOff mirror 401; fallback unreliable on 401 unreliable the canary gate (section 07) the outage the gate exists to prevent 05 Digests, manifests and the supply chainPrecision matters here, because half the traps on this page come from conflating these
-objects:
+Half the mistakes on this page came from conflating these objects:
-A tag is a mutable, human-readable pointer. Zero, one or many tags can
-reference one manifest.
-A manifest digest identifies exact manifest bytes. A multi-platform
-index has its own digest; each platform manifest has another; layers and
-config blobs have their own again.
+A tag is a mutable pointer; zero or many tags can reference one
+manifest.
+A manifest digest identifies exact manifest bytes; a multi-platform
+index , each platform manifest, and every layer and config blob carry their
+own.
name:tag@digest is client-side grammar, not wire protocol: the OCI
distribution spec takes a tag or a digest in the URL. Clients resolve the digest
-and ignore the tag (Kubernetes documents exactly this), so digest-pinned pulls are immune
-to tag moves - a fact this page once mislearned as a cache trap and later retested: the
-historical zot rejection of combined references did not reproduce on v2.1.17. Traps are
-perishable; retest your scars.Digest pinning provides content integrity - not provenance, not
authorisation, not availability, and not freedom from network fetches on a cold store.
-For a mirror, digests carry one sharp operational rule, learned here as a crashloop:
-zot converts Docker-schema manifests to OCI by default, and conversion changes the digest -
-silently breaking every digest pin and signature downstream. The pairing that prevents it
-(preserveDigest: true per upstream, with http.compat: ["docker2s2"])
-is mandatory in both directions: the registry refuses to start with one and not the other.
-This deployment runs both, on all five upstreams - observed in the live config.
+For a mirror, digests carry one sharp operational rule: zot converts Docker-schema
+manifests to OCI by default, and conversion changes the digest - breaking every digest pin
+and signature downstream. The pairing that prevents it (preserveDigest: true
+per upstream, with http.compat: ["docker2s2"]) is mandatory in both
+directions: the registry refuses to start with one and not the other (the Loop 1 field
+note is this rule being learned). This deployment runs both on all five upstreams.
-Preserving digests does not mean the supply chain travels whole. OCI 1.1
-referrers - signatures, SBOMs, attestations attached via the subject field and
-served by the referrers API - are separate objects with their own discovery path. A
-pull-through cache that mirrors manifests and blobs does not automatically carry them.
-Nothing here verifies signatures today; if it did, disconnected verification would also
-need the trust material carried locally (a public key, or a trusted-root bundle for a
-private signing stack). That is stated as a boundary, not an aspiration.
-
-THE HOMELAB CLAUSE
+Digest preservation and artifact completeness are different properties. The docs tie
+preserveDigest/compat to keeping mirrored manifest bytes and
+media types - and therefore signature validity - aligned with upstream. Whether
+signature, SBOM and attestation objects arrive is separate: OCI 1.1 referrers
+ride the subject field and their own API, legacy Cosign signatures ride
+tag-schema conventions, and both depend on origin support and this version's sync
+behaviour. This deployment has not verified a complete referrer graph for any origin and
+performs no cryptographic signature verification; disconnected verification would also
+need locally held trust material. Both gaps sit in the open register. Production estates
+promote releases by digest and decide per policy whether referrers travel too; this lab
+runs only the digest half.
06 Retention, GC and the digest-only trapCached bricks queued behind a jammed intake: availability problems here are quiet, not loud.
-This section corrects the largest error in this page's own first edition. I wrote:
-"no retention config means keep everything". That is true for tags, and dangerously false
-for everything else - and "everything else" includes precisely the content this mirror
-exists to hold.
-
-The verified semantics, from upstream documentation and the deployed version's source:
+This section corrects the largest error in this page's first edition, which said "no
+retention config means keep everything". True for tags; false for everything else - and
+"everything else" includes the content this mirror exists to hold. The semantics, from
+upstream documentation and the deployed version's source:
-With no retention configuration, all tags are retained - and
-all untagged manifests are deleted by garbage collection (unless referenced
-by an index or artifact) once they are older than gcDelay. This deployment's
-delay is the 1-hour default, with GC sweeping daily.
+With no retention configuration, all tags are retained and
+all untagged manifests are deleted by GC (unless referenced by an index
+or artifact). Timing is two-stage: the manifest ages past gcDelay (1h here),
+then deletion happens at the next sweep (gcInterval, 24h here) - the survival
+window depends on where creation falls relative to that sweep, not a fixed one-hour fuse.
Digest-only pull-through entries are stored as untagged manifests.
-That is the exact wording of the upstream fix's problem statement: pull an image through
-the mirror by digest, and the cache entry has no tag to protect it.
-SAY IT A digest-pinned fleet, pulling through a
-mirror whose GC treats digest-only entries as garbage: the cache evicts exactly what the
-estate is built on.
+Combined with section 04's revalidation behaviour: tag pulls are cached but
+contact upstream anyway; digest pulls serve locally but their entries are
+GC-eligible between sweeps - a digest-pinned fleet caches exactly the entry class GC may
+delete. On v2.1.17 this mirror is a rate-limit shield and a latency win, not yet a
+disconnection shelf ; the first edition implied otherwise and was wrong.
-Put the pieces together with section 04's revalidation fact and the deployed behaviour
-is this: tag pulls are cached but always phone upstream anyway; digest
-pulls are served locally but their cache entries are eligible for eviction within hours.
-On v2.1.17, this mirror is a rate-limit shield and a latency win - it is not yet a
-disconnection shelf , and the first edition of this page was wrong to imply
-otherwise.
-
-Diagram C · what keeps an object alive
+
Diagram C · what keeps an object alive
The OCI object graph and which references protect content from garbage collection
A tag points to an index; the index references platform manifests; manifests reference layers and a config blob. Referrers attach to a manifest via a subject field. A separate digest-only manifest sits with no tag pointing at it; it is eligible for garbage collection after the delay on the deployed version. Retention rules that keep tags do not protect the untagged manifest.
@@ -564,16 +537,16 @@ otherwise.
digest-only cached manifest NO tag references it
GC after gcDelay (1h here) deleteUntagged default: true
keepTags rules protect TAGS in their repo. Nothing on v2.1.17/18 protects the amber box;
- pull-aware keepUntagged ships in v2.1.19 (2026-08-04). Node-local caches can mask the eviction for days.
+ keepTags protects TAGS in its repo. On v2.1.17/18 the amber box has one blanket lever: deleteUntagged:false,
+ which keeps ALL untagged manifests (capacity trade). Selective pull-aware keepUntagged ships in v2.1.19.
green = protected by a tag · amber/red = the eviction path · dashed magenta = referrers (separate lifecycle)
-
The lesson: reachability, not existence, is what GC respects - and a
-pull-through cache full of digest pulls is a graveyard of unreachable-by-tag objects.
+
GC evaluates reference reachability: an untagged digest-only manifest can be
+deleted even while its blobs remain present, and node-local image stores can mask the
+eviction for days.
Retention policy semantics, stated precisely because two nearby systems use opposite
-rules:
+Retention policy semantics - two nearby systems use opposite matching rules:
Retention policies match per repository, first match wins - order the
@@ -589,24 +562,21 @@ match. Two adjacent config blocks, two opposite precedence rules. Label which on
reasoning about.
-Mitigations, honestly ranked: upgrade to v2.1.19+ and configure
-keepUntagged with pull-activity rules (the designed fix); until then, widen
-gcDelay/retention.delay so eviction pressure drops (the upstream
-maintainer's own interim suggestion), or set deleteUntagged: false and accept
-that the cache only grows - a capacity trade, not a free lunch. Disabling GC entirely
-trades eviction for guaranteed storage exhaustion with no reclaim path; it is listed here
-to be argued against.
+Mitigations, ranked: upgrade to v2.1.19+ for pull-aware
+keepUntagged (the designed fix, schema to be validated against the actual
+binary before rollout); until then, widen gcDelay/retention.delay
+(the upstream maintainer's interim suggestion), or set deleteUntagged: false,
+which protects every untagged manifest at the cost of unbounded cache growth. Disabling GC
+entirely swaps eviction for storage exhaustion with no reclaim path.
SAFE VALIDATION - PROPOSED, NOT YET RUN disposable zot
-instance only: record version + sanitised config; pull an image by digest through it;
-confirm on the registry side that the stored manifest is untagged; shorten GC timers (in
-the disposable instance only); observe the manifest before and after the sweep; then block
-upstream and repeat the pull from a clean runtime store, recording whether the mirror
-serves or re-fetches. Two warnings from upstream documentation: the retention verification
-tool executes orphan-blob GC for real even in dry-run, and on local storage it requires the
-registry stopped. Never point retention experiments at live storage.
+The eviction claim is documented upstream and consistent with this config; it has not been
+reproduced here. The reproduction, in a disposable instance only: record version
+and sanitised config; pull by digest; confirm the stored manifest is untagged; shorten GC
+timers; observe the manifest across a sweep; then block upstream and re-pull from a clean
+runtime store, recording serve vs re-fetch. Two upstream warnings: the retention
+verification tool executes orphan-blob GC for real even in dry-run, and local storage
+requires the registry stopped. Never point retention experiments at live storage.
07 The authentication migrationThe proving gate: a node counts as migrated when an authenticated pull succeeds where an anonymous one cannot.
-Target state: the mirror refuses anonymous pulls. Current state: anonymous read on, with
-per-node credentials staged in machine configs (inert until each node's reboot - the
-documented behaviour). The migration is a sequencing problem with one trap at its centre:
+Target state: the mirror refuses anonymous pulls. Current state: anonymous read on,
+per-node credentials staged inert in machine configs until each reboot. The sequencing
+problem has one central hazard: while anonymous read is on, an ordinary pull says
+nothing about auth - a node whose credentials never applied is served as an
+anonymous reader and false-passes the gate. The canary must live in a repository that
+denies anonymous read, so success can only mean an authenticated pull.
-While anonymous read is on, an ordinary pull proves nothing about auth.
-A node whose credentials never applied sends no authorisation header, gets served as an
-anonymous reader, and false-passes the gate. The canary must live in a repository that
-denies anonymous read, so a successful pull can only mean an authenticated pull.
-
-That design is now upstream-validated rather than assumed: zot authorisation resolves
-per-repository policies by longest match - the most specific path pattern
-wins, and ** is explicitly the default policy for anything unmatched. A
-canary/** entry that grants named identities read and carries no
-anonymousPolicy therefore denies anonymous on that path while the glob keeps
-the rest of the shelf open. Maintainer guidance confirms anonymous and authenticated access
-are evaluated independently. The staged policy:
+The design is upstream-supported: authorisation resolves per-repository policies by
+longest match (** is the default for anything unmatched), and
+maintainer guidance confirms anonymous and authenticated access evaluate independently. So
+a canary/** entry granting named identities read, with no
+anonymousPolicy, denies anonymous on that path while the glob keeps the rest
+of the shelf open. The staged policy:
"accessControl": {
"repositories": {
@@ -664,23 +631,20 @@ buildah, so the caveat is noted rather than felt.
Stage credentials in the node's machine config (inert; documented as requiring reboot).
Reboot the node at a planned window.
-Prove identity: pull the protected canary through the node's own
-runtime. Success = authenticated (anonymous cannot); failure = 401, back to step 1. The
-expected split, to be confirmed against observed responses when the test runs: 401 for
-missing/wrong credentials, 403 for a valid identity lacking the action - upstream docs
-state the 403 case explicitly only for OIDC identities, so the basic-auth 403 boundary is
-listed in the open-verification register rather than asserted.Prove interception separately: the canary pull is a direct reference
-to the mirror, so it cannot prove that docker.io/... references are being
-routed through the mirror at all. That second proof needs an original-upstream reference
-pulled on the node plus the mirror's request log showing it arrive - and a cold
-runtime store, because node-local content will satisfy the pull without any network and
-fake a pass.Only when every node passes both proofs does anonymousPolicy come off the
-glob. Rollback is the previous config commit, and it is named before the flip, not after.
+Identity check: pull the protected canary through the node's own
+runtime. Success = authenticated (anonymous cannot); 401 = back to step 1. The expected
+split - 401 for missing/wrong credentials, 403 for a valid identity lacking the action -
+sits in the open register, since the docs state the 403 case only for OIDC identities.Interception check, separately: the canary is a direct mirror
+reference, so it says nothing about whether docker.io/... references route
+through the mirror. That needs an original-upstream reference pulled on a node with a cold
+runtime store, correlated with the mirror's request log - node-local content would satisfy
+the pull without any network and fake a pass.Anonymous read comes off the glob only when every node passes both checks. Rollback is
+the previous config commit, named before the flip.
-Diagram E · the migration ladder
+
Diagram E · the migration ladder
The authentication migration ladder with its per-node proof gate and rollback point
Five stages: anonymous baseline, credentials staged inert, node reboot, the two-part proof - authenticated canary pull plus logged mirror interception - and the final anonymous-off flip with a named rollback commit. A failing node loops from the proof back to staging.
@@ -701,24 +665,27 @@ glob. Rollback is the previous config commit, and it is named before the flip, n
the flip is per-fleet; the proof is per-node - one unproven node under a fleet-wide flip is an outage wearing a green tick
-
Why two proofs: identity and interception fail independently, and each has a
-false-pass mode the other cannot detect.
+
Identity and interception fail independently; each has a false-pass mode
+the other cannot detect.
Behind the gate sit five separate trust domains, deliberately not shared: node pull
-credentials (machine config, reboot-bound), CI push credentials (human-held login), the
-mirror's own upstream credentials (one mounted secret, pod-restart-bound), TLS trust
-(estate wildcard, standard roots), and human admin access (SSO in front of the UI). The
-push credential carries this page's oldest scar: its hash and plaintext once drifted apart,
-rotation became impossible, and pushes stayed dead for twelve days until the rebuilt flow
-landed - three applications shipped the same day it came back. Hash and plaintext now live
-side by side, rotated together by a human, and the automation identity that syncs secrets
-is read-only by design so it can never half-rotate them again.
+Five trust domains never share material: node pull credentials (machine config,
+reboot-bound), the push credential (used by the operator-run build host for registry
+logins, not embedded in CI configuration), the mirror's upstream credentials (one mounted
+secret, pod-restart-bound), TLS trust, and human admin access (SSO in front of the UI).
+The push credential carries this page's oldest incident: in July 2026 its bcrypt hash and
+plaintext drifted apart in the secrets manager, rotation became impossible, and pushes
+stayed dead for twelve days - three applications shipped the day the rebuilt flow landed.
+Today the hash and plaintext are separate entries in the same secrets manager, rotated as
+a pair by a human; the registry consumes only the derived htpasswd file, each plaintext
+reaches only its consumer, and the sync identity is read-only so automation cannot
+half-rotate the pair. Residual cost, stated: one secrets-manager project holds both
+halves, so its compromise yields verifier and credential together - versioned,
+consumer-scoped secret objects remain a possible refinement.
08 Observability that means something"The registry is up" is the least useful sentence in this page. The worst incident in
-this mirror's life: an in-flight sync wedged, every pull from that upstream hung, and the
-health endpoint returned 200 throughout. The log line was almost poetic:
+The worst incident in this mirror's life: an in-flight sync wedged, every pull from
+that upstream hung, and the health endpoint returned 200 throughout:
Replay · observed 2026-07-07 · the stalled sync
"image already demanded, waiting on channel"
@@ -729,30 +696,27 @@ $ curl -sfS --max-time 10 -o /dev/null -w '%{http_code}\n' \
-H 'Accept: application/vnd.oci.image.index.v1+json'
200 <- THIS is "the mirror works"
-Honesty about that incident, upgraded by reading the source: the log line is
-coalescing by design - concurrent requests for one image join the first sync
-rather than duplicating it, and since the deployed version syncs run on a detached
-background context with a three-hour default timeout, surviving client disconnects.
-Waiters blocking on a genuinely stalled sync until that timeout is consistent with what we
-saw; the restart was a recovery action, and the root cause was never isolated. Two knobs
-this config does not currently set - sync maxRetries (disabled by default
-upstream) and a tighter syncTimeout - are the first things a recurrence should
-change.
+The deployed source refines that story: the log line is coalescing by design -
+concurrent requests join the first sync, which runs on a detached background context with
+a three-hour default timeout and survives client disconnects. Waiters blocking on a
+stalled sync until that timeout matches what we saw; the restart was recovery, and the
+root cause was never isolated. Two knobs this config leaves unset - sync
+maxRetries (disabled by default upstream) and a tighter
+syncTimeout - are the first change on recurrence.
-What this deployment actually exposes, verified against the running version's source:
-/livez, /readyz and /startupz exist as real health
-endpoints (absent from the docs at this version - a documentation gap, not an invention),
-alongside the spec's /v2/. The Prometheus metrics extension exists upstream
-(real series names include zot_http_requests_total,
-zot_repo_storage_bytes, zot_repo_downloads_total,
-zot_storage_lock_latency_seconds) - and is not enabled in this
-deployment . Wiring it, with scrape auth, is on the open list; no dashboard here
-pretends otherwise.
+From the running version's source: /livez, /readyz and
+/startupz exist as health endpoints (absent from this version's docs),
+alongside the spec's /v2/. The Prometheus metrics extension exists upstream -
+series names include zot_http_requests_total,
+zot_repo_storage_bytes, zot_repo_downloads_total and
+zot_storage_lock_latency_seconds - and is not enabled here ;
+wiring it, with scrape auth, is on the open list.
-The layered checks that would actually mean something, each answering one question:
+The layered checks, one question each:
-
-Check Question it answers Status here
+Layered checks - one question each
+Check Question it answers Status here /livez / /readyzis the process up / initialised available authenticated manifest GET of a local-only canary can the right identity read real content from local storage proposed anonymous GET of the canary expecting 401 is the protection actually protecting proposed certificate expiry, credential age what breaks on a schedule proposed
The test matrix for the caching claims - each cell is an experiment, not an assumption
-(status: the two digest rows are documented upstream and consistent with observed
-behaviour here; the disconnected rows have not been run in this lab):
+The caching test matrix (digest rows: source-verified and consistent with behaviour
+here; disconnected rows: not yet run in this lab):
-
-Scenario Expected on v2.1.17 What proves it
+Cache behaviour - each row is an experiment
+Scenario Expected on v2.1.17 What proves it cached tag, upstream reachable served; upstream contacted anyway (revalidation) zot log shows upstream request; latency includes round-trip cached tag, upstream unreachable fails or stalls until timeout, then local fallback path pull timing vs sync timeout; error text cached digest, upstream reachable served locally, no upstream contact absence of upstream request in zot logs during the pull A central curated source and site-local registries: each site holds its own shelf, so a cut WAN idles nothing.
-The first edition of this section asserted what "production" does. That was the wrong
-register: production chooses from patterns against requirements. The honest version:
+The first edition of this section asserted what "production" does; production chooses
+from patterns against requirements:
-Single registry, accepted failure domain - exactly this lab's shape.
-Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or to a paused
-deploy window (fallback off) and that cost is accepted in writing.Sync-based HA - upstream documents active/standby and active/active
-pairs of independent zot instances, each with its own storage, mirroring each
-other behind a load balancer; the documented caveat is the synchronisation window between
-polls, which bounds what a failover can lose.Scale-out clustering - upstream's other documented shape: instances
-shard repositories by hash and proxy to the owner, classically over shared S3-compatible
-storage. That is horizontal scale for load; the shared storage is then the availability
-story, which is a different property than the HA pair above. Naming which property you are
-buying is the design act.Edge estates - a central curated source promotes releases by digest;
-each site runs a small local registry holding its release set, synced on a schedule, so a
-severed WAN idles nothing at pull time. A disconnection claim is only valid when the
-required content, its referrers if policy demands them, trust roots, credentials and the
-registry's own bootstrap image are already local - the checklist from diagram D, applied
-per site.Single registry, accepted failure domain - this lab's shape.
+Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or a paused
+deploy window (fallback off), with the cost accepted in writing.Sync-based HA - documented active/standby or active/active pairs of
+independent instances, each with its own storage, mirroring each other behind a
+load balancer. The caveat: the synchronisation window between polls bounds what failover
+can lose.Scale-out clustering - instances shard repositories by hash and proxy
+to the owner, classically over shared S3-compatible storage. Horizontal scale for load;
+the shared storage becomes the availability story - a different property from the HA pair.Edge estates - a central source promotes releases by digest; each
+site runs a local registry holding its release set on a schedule, so a severed WAN idles
+nothing at pull time. A disconnection claim is valid only when content, required
+referrers, trust roots, credentials and the registry's own bootstrap image are already
+local (diagram D's right-hand column, per site).
-Whatever the topology: recovery time and recovery point get numbers before an incident
-provides them; backups are restore-tested copies off the failure domain (array snapshots
-and RAID protect against disks, not against the array, the site, or an operator error -
-they are inputs to a backup strategy, not the strategy); capacity is planned against the
-retention policy from section 06, because "how big does the cache get" is a policy output,
-not a guess; and the circular dependencies from section 04 are drawn per site, because
-every one of them exists at every scale - the only thing that changes is how expensive
-they are to ignore.
+Whatever the topology: recovery time and point objectives get numbers before an
+incident supplies them; backups are restore-tested copies off the failure domain (array
+snapshots and RAID protect against disk loss, not against the array, the site or operator
+error); cache capacity is planned from the section 06 retention policy; and the section 04
+circular dependencies get drawn per site, because each exists at every scale - only the
+cost of ignoring them changes.
10 Verified, open, and where the claims come fromProof of life is one authenticated byte transfer through the real path - not a status page.
-When a node powers on, this fleet's gate makes it pull one known, first-party image
-through its own runtime before it counts as a member. That gate is the whole page in
-miniature: prove the path, not the process.
+When a node powers on, the fleet's admission check performs an authenticated manifest
+pull through the node's own runtime before the node counts as a member: a process-health
+response validates neither routing nor authorisation. Most of this page condenses into
+that one command.
-
+
Architecture decisions
-Decision Chosen Why Trade-off Rollback / alternative Status Decision Chosen Why Trade-off Rollback / alternative Status Origin fallback ON (default kept) bootstrap survivability, Loop 1 silent mirror bypass possible skipFallback:true + preseed (airgap shape) observed Sync mode onDemand, all upstreams cache follows real usage; Hub-safe tag pulls revalidate upstream; digest entries untagged polled sync for a curated release set observed Digest preservation preserveDigest + docker2s2, x5 digest pins + signatures survive mirroring mandatory config pairing (crashloop scar) none - non-negotiable for digest-pinned estates observed