projects: add The Mirror pull-through registry case study; wire cover art for both learn-site projects
build-and-deploy / build (push) Failing after 12m49s
build-and-deploy / build (push) Failing after 12m49s
Adds mirror-registry as a featured project (order 26, beside The Exploded Cluster) covering the zot deployment, the bootstrap and auth circular dependencies, and the retention/revalidation behaviours that corrected the first published write-up. Adds the cover: field to exploded-cluster (it had none, so the card rendered imageless) and stages public/covers/image.md with the generation prompts and processing steps for both covers.
This commit is contained in:
@@ -4,9 +4,10 @@ outcome: "A scroll-driven teaching site that takes container platforms apart one
|
||||
summary: "A single-page course on how Kubernetes and OpenShift delivery actually works - each subject drawn as one exploded illustration, sliced into its real components and wired to the scroll, then shipped through the same GitOps and CSP discipline as the rest of the estate."
|
||||
role: "Author / Engineer"
|
||||
period: "2026"
|
||||
stack: ["Astro-free static build", "GSAP ScrollTrigger", "Strict CSP", "Cloudflare Tunnel", "Argo CD", "zot mirror"]
|
||||
stack: ["Dependency-free static build", "GSAP ScrollTrigger", "Strict CSP", "Cloudflare Tunnel", "Argo CD", "zot mirror"]
|
||||
featured: true
|
||||
order: 25
|
||||
cover: "exploded-cluster"
|
||||
links:
|
||||
- label: "learn.bztmon.com"
|
||||
href: "https://learn.bztmon.com"
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: "The Mirror: A Pull-Through Registry"
|
||||
outcome: "Ended anonymous registry rate-limits across a six-node fleet, then published the failure modes the deployment revealed - including the one that contradicted my own first write-up."
|
||||
summary: "A zot OCI registry fronting five upstreams for a Talos/containerd fleet: on-demand pull-through, digest preservation, a staged authentication rollout with a per-node proof gate, and a documented set of behaviours - garbage collection, cache revalidation, upstream routing - that only surfaced when the deployment was read against the source rather than the marketing."
|
||||
role: "Author / Engineer"
|
||||
period: "2026"
|
||||
stack: ["zot v2.1.17", "Talos Linux", "containerd", "OCI distribution", "Argo CD", "External Secrets"]
|
||||
featured: true
|
||||
order: 26
|
||||
cover: "mirror-registry"
|
||||
links:
|
||||
- label: "learn.bztmon.com/mirror"
|
||||
href: "https://learn.bztmon.com/mirror"
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
A CI build stalled with no visible error. The cause was an anonymous Docker Hub pull hitting
|
||||
the rate limit and answering with HTTP 429 - which a retry wrapper swallowed, four outer
|
||||
attempts around a builder doing up to four of its own, so one failing pull could generate up
|
||||
to sixteen requests against the limit it was trying to outlast. Beneath that specific bug sat
|
||||
the structural one: every node and every build asked the public internet for the same bytes,
|
||||
repeatedly and forever.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **The registry is also a dependency of the thing that runs it.** Its own image lives on a
|
||||
registry it mirrors, and the node hosting it boots its workloads through it.
|
||||
- **Digest-pinned estate.** Every deployment references images by digest, so anything that
|
||||
rewrites a manifest in transit breaks the pins and any signature attached to them.
|
||||
- **Authentication cannot flip in one step.** Node registry credentials only apply after a
|
||||
reboot, and a runtime whose auth is wrong does not degrade gracefully - it stops pulling.
|
||||
|
||||
## Design
|
||||
|
||||
A single zot instance fronts five upstream registries in on-demand pull-through mode: a miss
|
||||
fetches and caches, a hit serves locally. Digest preservation is enabled per upstream, paired
|
||||
with the media-type compatibility flag the registry refuses to start without - conversion to
|
||||
OCI would otherwise change every mirrored manifest digest.
|
||||
|
||||
Consumers point at it through Talos machine config. The mirror is the only listed endpoint,
|
||||
but it is not a hard dependency, because the platform falls back to the origin registry
|
||||
implicitly unless that behaviour is disabled. That default is what breaks the bootstrap
|
||||
circle: when the hosting node boots and the mirror is not yet running, the pull falls through
|
||||
and the registry's own image arrives from upstream. The fleet's recovery tooling deliberately
|
||||
does not use the mirror at all - its image is cached outside the cluster, so it stays
|
||||
available when the platform it repairs is not.
|
||||
|
||||
Authentication is a staged rollout rather than a flip. Anonymous read stays on while each node
|
||||
receives credentials and reboots, and a node only counts as migrated once it passes two
|
||||
separate checks: an authenticated pull of a canary repository that denies anonymous access
|
||||
(so success cannot come from being served as an anonymous reader), and a separate correlation
|
||||
between an original-upstream reference pulled on a cold node and the mirror's own request log
|
||||
(because the first check proves identity, not routing).
|
||||
|
||||
## What reading the source changed
|
||||
|
||||
The first published version of this write-up was wrong in a way worth keeping on the record.
|
||||
I had written that with no retention policy configured, the cache keeps everything. That holds
|
||||
for tagged images. It does not hold for untagged manifests, which garbage collection removes
|
||||
once they age past the delay - and digest-only pull-through entries are stored untagged. A
|
||||
digest-pinned fleet pulling through this mirror therefore caches exactly the class of object
|
||||
its garbage collector is entitled to delete. Selective, pull-aware retention for untagged
|
||||
content exists upstream, but only from a release later than the one deployed here; until an
|
||||
upgrade, the levers are a wider delay window or a blanket policy that trades eviction for
|
||||
unbounded growth.
|
||||
|
||||
Two related behaviours came from the same exercise. Pulls by tag contact the upstream registry
|
||||
even when the content is already cached, so a warm cache is not equivalent to independence
|
||||
from the origin. And five upstreams currently share one flat namespace, which works because
|
||||
their repository paths happen to be largely disjoint - a probability rather than a design, with
|
||||
per-origin prefixes as the fix.
|
||||
|
||||
## Security and reliability decisions
|
||||
|
||||
- **Fallback is a recorded policy choice.** Keeping it enabled buys bootstrap survivability
|
||||
and availability during mirror outages, at the cost of pulls bypassing the mirror unnoticed
|
||||
and reaching origin limits. A regulated or disconnected estate makes the opposite call
|
||||
deliberately, along with preseeded content and a tested recovery path.
|
||||
- **Credentials are separated by consumer.** Node pull credentials, the build host's push
|
||||
credential, and the registry's own upstream credentials never share material, and the
|
||||
automation identity that synchronises secrets into the cluster is read-only so it cannot
|
||||
half-rotate a credential pair - the failure mode behind a twelve-day outage on this estate
|
||||
when a stored hash and its plaintext drifted apart.
|
||||
- **Health checks that test the path.** A process returning 200 while every pull hangs is a
|
||||
state this deployment has actually been in. The checks that matter authenticate, request a
|
||||
real manifest, and confirm the request arrived through the mirror.
|
||||
|
||||
## Outcome
|
||||
|
||||
Fleet-wide pulls resolve locally instead of hitting public rate limits, and the estate's
|
||||
disconnected posture is now honestly bounded: the mirror is a rate-limit shield and latency
|
||||
win today, not yet a shelf that survives a severed link. The full teardown - configuration,
|
||||
diagrams, failure-mode matrix, and the tests still outstanding - is published as a public
|
||||
case study.
|
||||
|
||||
## Future improvements
|
||||
|
||||
Upgrade to a release with selective untagged retention and write the policy before the
|
||||
upgrade; per-origin namespace prefixes to make upstream collisions impossible rather than
|
||||
improbable; enable the metrics extension so cache growth and eviction are visible; and run
|
||||
the disconnected drill that would let the site claim WAN independence with evidence.
|
||||
Reference in New Issue
Block a user