projects: add The Mirror pull-through registry case study; wire cover art for both learn-site projects
build-and-deploy / build (push) Failing after 12m49s

Adds mirror-registry as a featured project (order 26, beside The Exploded
Cluster) covering the zot deployment, the bootstrap and auth circular
dependencies, and the retention/revalidation behaviours that corrected the
first published write-up. Adds the cover: field to exploded-cluster (it had
none, so the card rendered imageless) and stages public/covers/image.md with
the generation prompts and processing steps for both covers.
This commit is contained in:
2026-08-25 09:18:11 +10:00
parent 8d394fe2c7
commit c5b1044d7a
3 changed files with 202 additions and 1 deletions
+2 -1
View File
@@ -4,9 +4,10 @@ outcome: "A scroll-driven teaching site that takes container platforms apart one
summary: "A single-page course on how Kubernetes and OpenShift delivery actually works - each subject drawn as one exploded illustration, sliced into its real components and wired to the scroll, then shipped through the same GitOps and CSP discipline as the rest of the estate."
role: "Author / Engineer"
period: "2026"
stack: ["Astro-free static build", "GSAP ScrollTrigger", "Strict CSP", "Cloudflare Tunnel", "Argo CD", "zot mirror"]
stack: ["Dependency-free static build", "GSAP ScrollTrigger", "Strict CSP", "Cloudflare Tunnel", "Argo CD", "zot mirror"]
featured: true
order: 25
cover: "exploded-cluster"
links:
- label: "learn.bztmon.com"
href: "https://learn.bztmon.com"
+102
View File
@@ -0,0 +1,102 @@
---
title: "The Mirror: A Pull-Through Registry"
outcome: "Ended anonymous registry rate-limits across a six-node fleet, then published the failure modes the deployment revealed - including the one that contradicted my own first write-up."
summary: "A zot OCI registry fronting five upstreams for a Talos/containerd fleet: on-demand pull-through, digest preservation, a staged authentication rollout with a per-node proof gate, and a documented set of behaviours - garbage collection, cache revalidation, upstream routing - that only surfaced when the deployment was read against the source rather than the marketing."
role: "Author / Engineer"
period: "2026"
stack: ["zot v2.1.17", "Talos Linux", "containerd", "OCI distribution", "Argo CD", "External Secrets"]
featured: true
order: 26
cover: "mirror-registry"
links:
- label: "learn.bztmon.com/mirror"
href: "https://learn.bztmon.com/mirror"
---
## Problem
A CI build stalled with no visible error. The cause was an anonymous Docker Hub pull hitting
the rate limit and answering with HTTP 429 - which a retry wrapper swallowed, four outer
attempts around a builder doing up to four of its own, so one failing pull could generate up
to sixteen requests against the limit it was trying to outlast. Beneath that specific bug sat
the structural one: every node and every build asked the public internet for the same bytes,
repeatedly and forever.
## Constraints
- **The registry is also a dependency of the thing that runs it.** Its own image lives on a
registry it mirrors, and the node hosting it boots its workloads through it.
- **Digest-pinned estate.** Every deployment references images by digest, so anything that
rewrites a manifest in transit breaks the pins and any signature attached to them.
- **Authentication cannot flip in one step.** Node registry credentials only apply after a
reboot, and a runtime whose auth is wrong does not degrade gracefully - it stops pulling.
## Design
A single zot instance fronts five upstream registries in on-demand pull-through mode: a miss
fetches and caches, a hit serves locally. Digest preservation is enabled per upstream, paired
with the media-type compatibility flag the registry refuses to start without - conversion to
OCI would otherwise change every mirrored manifest digest.
Consumers point at it through Talos machine config. The mirror is the only listed endpoint,
but it is not a hard dependency, because the platform falls back to the origin registry
implicitly unless that behaviour is disabled. That default is what breaks the bootstrap
circle: when the hosting node boots and the mirror is not yet running, the pull falls through
and the registry's own image arrives from upstream. The fleet's recovery tooling deliberately
does not use the mirror at all - its image is cached outside the cluster, so it stays
available when the platform it repairs is not.
Authentication is a staged rollout rather than a flip. Anonymous read stays on while each node
receives credentials and reboots, and a node only counts as migrated once it passes two
separate checks: an authenticated pull of a canary repository that denies anonymous access
(so success cannot come from being served as an anonymous reader), and a separate correlation
between an original-upstream reference pulled on a cold node and the mirror's own request log
(because the first check proves identity, not routing).
## What reading the source changed
The first published version of this write-up was wrong in a way worth keeping on the record.
I had written that with no retention policy configured, the cache keeps everything. That holds
for tagged images. It does not hold for untagged manifests, which garbage collection removes
once they age past the delay - and digest-only pull-through entries are stored untagged. A
digest-pinned fleet pulling through this mirror therefore caches exactly the class of object
its garbage collector is entitled to delete. Selective, pull-aware retention for untagged
content exists upstream, but only from a release later than the one deployed here; until an
upgrade, the levers are a wider delay window or a blanket policy that trades eviction for
unbounded growth.
Two related behaviours came from the same exercise. Pulls by tag contact the upstream registry
even when the content is already cached, so a warm cache is not equivalent to independence
from the origin. And five upstreams currently share one flat namespace, which works because
their repository paths happen to be largely disjoint - a probability rather than a design, with
per-origin prefixes as the fix.
## Security and reliability decisions
- **Fallback is a recorded policy choice.** Keeping it enabled buys bootstrap survivability
and availability during mirror outages, at the cost of pulls bypassing the mirror unnoticed
and reaching origin limits. A regulated or disconnected estate makes the opposite call
deliberately, along with preseeded content and a tested recovery path.
- **Credentials are separated by consumer.** Node pull credentials, the build host's push
credential, and the registry's own upstream credentials never share material, and the
automation identity that synchronises secrets into the cluster is read-only so it cannot
half-rotate a credential pair - the failure mode behind a twelve-day outage on this estate
when a stored hash and its plaintext drifted apart.
- **Health checks that test the path.** A process returning 200 while every pull hangs is a
state this deployment has actually been in. The checks that matter authenticate, request a
real manifest, and confirm the request arrived through the mirror.
## Outcome
Fleet-wide pulls resolve locally instead of hitting public rate limits, and the estate's
disconnected posture is now honestly bounded: the mirror is a rate-limit shield and latency
win today, not yet a shelf that survives a severed link. The full teardown - configuration,
diagrams, failure-mode matrix, and the tests still outstanding - is published as a public
case study.
## Future improvements
Upgrade to a release with selective untagged retention and write the policy before the
upgrade; per-origin namespace prefixes to make upstream collisions impossible rather than
improbable; enable the metrics extension so cache growth and eviction are visible; and run
the disconnected drill that would let the site claim WAN independence with evidence.