jwright/exploded-cluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

mirror second-pass close-out: retention diagram precision, evidence taxonomy, sourced Talos reboot claim, qualified failure matrix, Hub pull terminology, referrer completeness split, storage semantics, secret-rotation truth, table captions + scope + keyboard regions, all-lazy plates, en-AU, editorial reduction, extended gates

Browse Source
This commit is contained in:
Jonathon Wright
2026-08-25 08:52:35 +10:00
parent 7f7035b98f
commit d443f9ad6b
6 changed files with 312 additions and 340 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dockerfile
+5 -5
View File
@@ -1,11 +1,11 @@
# syntax=docker/dockerfile:1
# Static teaching site: bash assembles dist/ from the split source, then a pinned
# nginx-unprivileged serves it. No Node build — build.sh is plain bash + coreutils.
# Static teaching site: build.sh runs the node assembler (pilot/assemble-arc.mjs) to
# produce dist/, then a pinned nginx-unprivileged serves it. No package installs - the
# assembler is dependency-free node.
# ---- build stage ----------------------------------------------------------
# Reusing the digest the site image already pins: it's a bookworm-slim base that is
# verified and cached in the zot pull-through mirror, so this layer costs nothing
# extra. Only bash/sed/find/install are used here — node itself is incidental.
# A bookworm-slim node base, digest-pinned and typically served from the zot
# pull-through mirror; node runs the assembler.
FROM node:22-bookworm-slim@sha256:d9f850096136edbc402debdd8729579a288aac64574ada0ff4db26b6ae58b0b2 AS build
ARG BUILD_REV=dev
ENV BUILD_REV=$BUILD_REV
README.md
+2 -2
View File
@@ -1,7 +1,7 @@
# The Teaching Lab (learn.bztmon.com)
A multi-page static technical learning site: scroll-driven teardowns of infrastructure the
author actually operates. Dark engineering-dossier aesthetic (void `#070b14`, cyan `#3fbaf5`,
A static learning site documenting infrastructure I operate in my homelab - the decisions,
the failures, and the validation still outstanding. Dark engineering-dossier aesthetic (void `#070b14`, cyan `#3fbaf5`,
magenta `#e879f9`), dependency-light by design, strict security posture.
## Pages
pilot/assemble-arc.mjs
+16 -5
View File
@@ -141,7 +141,7 @@ const OG_IMG = SITE + heroUrls["__HERO_GITOPS__"];
// shared document shell: every page on the site gets the same head discipline
const shell = ({ title, desc, path, ogImg, scripts, bodyHtml }) => `<!doctype html>
<html lang="en">
<html lang="en-AU">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
@@ -178,7 +178,7 @@ const html = shell({
});
// ---- The Mirror (chapter 02): fingerprint its plates + its app script -------
const MIRROR_DESC = "A pull-through container registry taken apart: why a lab runs its own mirror, the chicken-and-egg loops inside it, and the scars that taught each rule.";
const MIRROR_DESC = "How a zot pull-through registry behaves under Talos and containerd: upstream routing, digest preservation, garbage collection, authentication rollout, failure modes, and the lab evidence behind each decision.";
let mirror = readFileSync(join(ROOT, "pilot/mirror.tpl.html"), "utf8");
const mirrorPlates = {};
for (const n of ["20", "21", "22", "23", "24", "25"]) {
@@ -186,10 +186,10 @@ for (const n of ["20", "21", "22", "23", "24", "25"]) {
mirrorPlates[n] = url;
const dims = jpegSize(buf);
if (!dims) throw new Error(`mirror plate ${n}: JPEG dimensions unreadable`);
// plate 20 is the page's first image (near the fold) -> eager/LCP; the rest lazy.
const load = n === "20" ? "eager" : "lazy";
// every plate sits below the fold (fact panel + contents precede the first scene),
// so all six lazy-load; intrinsic dims come from the actual bytes.
mirror = mirror.replace(`__M${n}__`,
`${url}" width="${dims.w}" height="${dims.h}" decoding="async" loading="${load}`);
`${url}" width="${dims.w}" height="${dims.h}" decoding="async" loading="lazy`);
}
const mirrorJs = readFileSync(join(ROOT, "pilot/mirror.js"));
const mirrorJsName = `mirror.${fp(mirrorJs)}.js`;
@@ -240,6 +240,11 @@ for (const [pg, doc] of [["cluster", html], ["mirror", mirrorHtml], ["index", di
// target the assertive phrasings and internal production jargon, not the topic words.
const mirrorBanned = [
[/never leaves the building/i, "retired absolute cache claim"],
[/SAY IT/, "removed rhetorical block resurfaced"],
[/HOMELAB CLAUSE/, "removed label resurfaced"],
[/Nothing on v2\.1\.17\/18 protects/, "over-broad retention annotation resurfaced"],
[/100 manifest requests per six hours/, "imprecise Hub limit phrasing"],
[/a tag is a suggestion, a digest is a fact/i, "retired slogan"],
[/up to twelve|twelve tries|12 attempts/i, "stale retry arithmetic (correct figure: sixteen)"],
[/plate-level/i, "internal production note leaked to readers"],
[/&middot; motion:/i, "art-direction caption leaked to readers"],
@@ -249,6 +254,9 @@ const mirrorBanned = [
for (const [re, msg] of mirrorBanned) if (re.test(mirrorHtml)) fail.push("mirror: " + msg);
const mirrorRequired = [
["zotregistry.dev", "primary zot references missing"],
["docs.docker.com/docker-hub/usage/pulls", "Docker Hub pulls reference missing"],
["v1alpha1", "Talos auth-reboot source link missing"],
["deleteUntagged", "blanket untagged lever discussion missing"],
["keepUntagged", "version-gated retention discussion missing"],
["/livez", "real health endpoints missing"],
['id="refs"', "references section missing"],
@@ -257,6 +265,9 @@ const mirrorRequired = [
];
for (const [needle, msg] of mirrorRequired) if (!mirrorHtml.includes(needle)) fail.push("mirror: " + msg);
must((mirrorHtml.match(/width="1600" height="872"/g) || []).length === 6, "mirror: expected six correctly-dimensioned plates");
must((mirrorHtml.match(/loading="lazy"/g) || []).length === 6, "mirror: expected all six plates lazy-loaded");
must((mirrorHtml.match(/<caption/g) || []).length === 5, "mirror: expected five table captions");
must(!/<th>(?!<)/.test(mirrorHtml.replace(/<th scope="col">/g, "")), "mirror: th without scope=col");
must(!/100vw/.test(mirrorHtml), "mirror: viewport-width unit reintroduces scrollbar overflow");
must(!mirrorHtml.includes("gsap"), "mirror: unexpected animation library reference");
pilot/index.tpl.html
+4 -4
View File
@@ -61,13 +61,13 @@ footer a{color:var(--cyan)}
</a>
<a class="card" href="/mirror">
<span class="thumb"><img src="__THUMB_MIRROR__" alt="" width="1600" height="873" loading="eager" decoding="async"></span>
<span class="thumb"><img src="__THUMB_MIRROR__" alt="" width="1600" height="872" loading="eager" decoding="async"></span>
<span class="meta">
<span class="k">Chapter 02 &middot; Dossier 005</span>
<h2>The Mirror <span class="arrow">&rarr;</span></h2>
<p class="outcome"><b>Outcome</b> &mdash; a pull-through container registry taken apart: why a
lab runs its own mirror, the four chicken-and-egg loops hiding inside it, and the scars that
taught each rule - told with the receipts.</p>
<p class="outcome"><b>Outcome</b> &mdash; how a zot pull-through registry behaves under Talos and
containerd: upstream routing, digest preservation, garbage collection, authentication
rollout and failure modes, with the lab evidence behind each decision.</p>
</span>
</a>
pilot/mirror.js
+1
View File
@@ -29,6 +29,7 @@
if (hit) visible[hit[0]] = e.isIntersecting;
});
if (visible.s23) startPulse(); // (re)arm the breather when its scene returns
onScroll(); // anchor jumps and observer-only transitions update without a scroll event
}, { rootMargin: "10% 0px" });
watched.forEach(function (w) { io.observe(w[1]); });
pilot/mirror.tpl.html
+284 -324
View File
@@ -37,9 +37,7 @@
.scene img{width:100%;height:auto;display:block;border-radius:.4rem}
.scene .cap{font-family:var(--mono);font-size:.74rem;color:var(--ink-dim);
letter-spacing:.1em;text-align:center;margin-top:.7rem;padding:0 1rem}
.sayit{border-left:3px solid var(--magenta);margin:2rem 0;padding:.4rem 0 .4rem 1.1rem;
font-size:1.26rem;font-weight:650;line-height:1.35;color:#fff}
.sayit .k{display:block;font-family:var(--mono);font-size:.7rem;letter-spacing:.2em;color:var(--magenta);margin-bottom:.3rem;font-weight:500}
[tabindex="0"]:focus-visible{outline:2px solid var(--cyan);outline-offset:3px}
.note{background:var(--card);border:1px solid var(--line);border-left:3px solid var(--cyan);
border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
.note .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan)}
@@ -114,11 +112,10 @@
<p class="backlink"><a href="/">&larr; The Teaching Lab</a></p>
<p class="eyebrow">Dossier / 005 &middot; operations</p>
<h1>The Mirror</h1>
<p class="outcome"><b>Outcome</b> &mdash; a pull-through OCI registry taken apart: why this
lab runs its own mirror, what the deployed version actually guarantees, the retention trap
that quietly evicts digest-pinned content, and how each lesson translates into production
platform engineering. Claims are labelled: observed here, documented upstream, or not yet
verified.</p>
<p class="outcome"><b>Outcome</b> &mdash; a pull-through OCI registry taken apart: what
the deployed version guarantees, the retention behaviour that evicts digest-pinned
content, and how each lesson translates to production. Claims carry evidence labels
throughout.</p>
</header>
<div class="facts">
@@ -129,9 +126,16 @@
<dt>Storage</dt><dd>50Gi local PVC; dedupe on, GC on (1h delay / 24h interval); no retention config</dd>
<dt>Auth state</dt><dd>anonymous read enabled; per-node pull credentials staged, not yet enforced</dd>
<dt>Fallback</dt><dd>origin fallback ON (platform default; deliberately not disabled)</dd>
<dt>Validated</dt><dd>2026-08-25 (live config read + upstream doc/source verification)</dd>
<dt>Open items</dt><dd>digest-entry retention (needs v2.1.19+), auth canary negative test, upstream namespace prefixes, disconnected-serve drill</dd>
<dt>Reviewed</dt><dd>2026-08-25: live cluster configuration read; upstream docs and v2.1.17 source checked; live runtime tests listed separately in the open register</dd>
<dt>Open items</dt><dd>pull-aware digest-entry retention (needs v2.1.19+; blanket alternatives are capacity-heavy), auth canary negative test, upstream namespace prefixes, disconnected-serve drill</dd>
</dl>
<p style="margin:.9rem 0 0;font-size:.8rem;color:var(--ink-dim);font-family:var(--mono)">
Evidence labels used below: <span class="st ok">lab</span> reproduced here with a dated receipt &middot;
<span class="st ok">config</span> present in the inspected config &middot;
<span class="st ok">source</span> established from the deployed version's source &middot;
<span class="st ok">docs</span> stated in version-appropriate documentation &middot;
<span class="st open">reported</span> described in an unresolved upstream issue &middot;
<span class="st prop">proposed</span> not deployed &middot; <span class="st open">open</span> not yet run</p>
</div>
<nav class="toc" aria-labelledby="toc-h">
@@ -157,24 +161,27 @@
<div class="cap">The mirror as a machine: upstream intakes, one cache shelf, one serving nozzle.</div>
</div></div>
<p>Nobody sets out to run a registry. You inherit the need the day a CI build stalls for no
reason you can see.</p>
<p>I added zot after a CI pull hit Docker Hub's anonymous rate limit and the retry wrapper
hid the 429. At the time of the incident, this site's build-stage pull was anonymous.</p>
<p>This site's own build pulls one base image from Docker Hub, anonymously. One evening the
build sat there - no visible error, no progress. The accurate version of that story: Docker
Hub does not go silent when it limits you. It answers with an explicit HTTP 429 and the error
code <code>toomanyrequests</code>; under the current published limits, anonymous pulls get
100 manifest requests per six hours per IPv4 address, and the accounting is per manifest
<code>GET</code>. The <em>silence</em> was manufactured on my side, by retries swallowing
that answer.</p>
<p>Docker Hub does not go silent when it limits you. It answers with HTTP 429 and the error
code <code>toomanyrequests</code>. Under the
<a href="https://docs.docker.com/docker-hub/usage/pulls/">current published limits</a>,
unauthenticated users get 100 pulls per six hours per IPv4 address (or IPv6 /64); a
single-platform image counts as one pull, a multi-architecture image counts once per
architecture pulled, version checks do not count, and a <code>HEAD</code> request can read
the rate-limit headers without consuming a pull. A separate abuse limiter covers all
request types with its own 429 form. The <em>silence</em> in this incident was
manufactured on my side, by retries swallowing the answer.</p>
<p>The swallower, recovered from the build script's git history, was a double retry stack: an
outer shell wrapper allowing four attempts, wrapped around <code>buildah build --retry 3</code>.
Buildah's flag counts <em>retries</em>, not attempts - one initial try plus up to three
retries, and it applies to registry push/pull operations. Four outer attempts, each
containing up to four inner attempts: <strong>up to sixteen requests</strong> for one
consistently failing pull, each one billed against the same rate limit it was trying to
outlast. Retry amplification turns a throttle into an outage and hides the evidence.</p>
<p>The swallower, recovered from the build script's git history, was a double retry stack:
an outer shell wrapper allowing four attempts, wrapped around
<code>buildah build --retry 3</code>. Buildah's flag counts <em>retries</em> - one initial
try plus up to three more, applying to registry push/pull operations. Four outer attempts,
each containing up to four inner attempts: <strong>up to sixteen requests</strong> for one
repeatedly failing registry operation (a build with several pulls can produce more), each
billed against the limit it was trying to outlast. Retry amplification turns a throttle
into an outage and hides the evidence while doing it.</p>
<div class="pane-k">Receipt &middot; the retry stack, as committed to the repo</div>
<pre class="pane">retry() {
@@ -188,30 +195,21 @@ retry buildah build --retry 3 --retry-delay 5s ...
<span class="cm"># --retry 3 = one attempt + three retries; the wrapper multiplies that by four.</span>
<span class="cm"># A 429 answered every one of them, and the backoffs read as a hang.</span></pre>
<p>The immediate fix was to pin the base image by digest - which stops tag re-resolution, but
does not remove the network round-trip: an ephemeral runner with a cold local store still
fetches the manifest behind that digest every build. The durable answer is structural:</p>
<div class="sayit"><span class="k">SAY IT</span>Why does a fleet of machines ask the public
internet for the same bytes, hundreds of times, forever?</div>
<p>A production platform answers with a mirror: one registry that fetches once and serves the
estate. The lab built one. This page takes it apart - including the parts that turned out
not to work the way I first believed.</p>
<p>Pinning the base image by digest stopped the tag re-resolution but not the round-trip:
an ephemeral runner with a cold store still fetches the manifest behind that digest every
build. The repeated origin pulls justified a shared cache. This page takes it apart,
including the parts that turned out not to work the way I first believed.</p>
<div class="note"><span class="k">FIELD NOTE</span><br>The mirror did not fix the red X in
CI. That red was a separate upstream bug in the CI system's log-finalise step - cosmetic,
tolerated, documented. Two problems, one symptom. Diagnose them separately or you will fix
the wrong one and declare victory.</div>
tolerated, documented. Two problems, one symptom; diagnose them separately.</div>
<h2 id="anatomy"><span class="n" aria-hidden="true">02</span>What the mirror actually does</h2>
<p>The mirror is zot: a single OCI registry in its own namespace, one 50Gi cache volume,
running the same digest-pinned deployment discipline as everything it serves. It fronts five
public registries in <strong>on-demand pull-through</strong> mode: a miss fetches from
upstream and caches; a hit serves from the shelf. zot also supports polled mirroring
(periodic full sync of matching content) and pre-seeding; Docker Hub is on-demand only -
upstream documentation is explicit that polled mirroring should not be pointed at Hub.</p>
<p>The mirror is zot: one registry, one 50Gi cache volume, digest-pinned like everything
it serves. Five public registries front it in <strong>on-demand pull-through</strong> mode -
a miss fetches and caches, a hit serves from the shelf. zot also offers polled mirroring
and pre-seeding; Docker Hub is on-demand only, per its documentation.</p>
<p>The storage block, from the live config:</p>
@@ -224,12 +222,16 @@ upstream documentation is explicit that polled mirroring should not be pointed a
"gcInterval": "24h"
}</pre>
<p><code>commit</code> fsyncs writes before acknowledging - crash safety. <code>dedupe</code>
hard-links identical blobs, which pays for itself when five upstreams ship the same base
layers under different names. And <code>gc</code> with its two timers looks innocuous here;
section 06 is about why it is the most consequential block on this page.</p>
<p><code>commit</code> asks zot to commit writes to disk immediately instead of relying on
buffered flushing; it narrows the buffered-write window, while end-to-end power-loss
durability still depends on the filesystem, volume and disk. <code>dedupe</code> uses hard
links on local filesystem storage (remote backends implement it differently), saving
capacity when five upstreams ship the same base layers under different names - at the cost
of a startup reconciliation pass that matters operationally when toggled on existing data.
And <code>gc</code> with its two timers looks innocuous here; section 06 is about why it is
the most consequential block on this page.</p>
<div class="diagram"><span class="k">Diagram A &middot; system context</span>
<div class="diagram" role="region" tabindex="0" aria-label="System context diagram"><span class="k">Diagram A &middot; system context</span>
<svg viewBox="0 0 860 400" role="img" aria-labelledby="da-t da-d">
<title id="da-t">System context: consumers, the mirror, its storage, five upstreams, and the trust boundaries between them</title>
<desc id="da-d">CI and Kubernetes nodes pull from the zot mirror over the LAN. The mirror stores content on a 50Gi volume and syncs on demand from five upstream registries across the internet boundary. Anonymous read is currently allowed inbound; outbound upstream credentials live in one mounted secret. The origin-fallback path from nodes directly to upstreams is shown dashed.</desc>
@@ -261,10 +263,9 @@ section 06 is about why it is the most consequential block on this page.</p>
</g>
</svg>
<p class="legend">solid = normal pull path &middot; dashed cyan = on-demand sync on miss &middot; dashed magenta = origin fallback</p>
<p class="why">The decision this explains: one cache serves two consumer classes, and the
two credential domains (nodes-to-mirror, mirror-to-upstreams) never mix.</p></div>
<p class="why">One cache, two consumer classes; the two credential domains never mix.</p></div>
<p>Consumers reach the mirror through the platform's machine-level registry config:</p>
<p>Consumers point at it via the platform's machine-level registry config:</p>
<pre>machine:
registries:
@@ -276,52 +277,44 @@ two credential domains (nodes-to-mirror, mirror-to-upstreams) never mix.</p></di
endpoints:
- https://zot.bztmon.org</pre>
<p>The mirror is the only listed endpoint, and that is still not a hard dependency, because
Talos documents an implicit final fallback: endpoints are tried in order, "and by default
the last implicit endpoint is the original upstream registry", unless
<code>skipFallback: true</code> says otherwise. The lab's first version listed the upstream
as an explicit second endpoint; the refinement that removed it came from reading the
behaviour properly - the default already guaranteed it. Know which of your safety nets you
built, and which are defaults you merely have not broken.</p>
<p>The mirror is the only listed endpoint, yet not a hard dependency: Talos tries
endpoints in order "and by default the last implicit endpoint is the original upstream
registry", unless <code>skipFallback: true</code>. An early version listed the upstream
explicitly as endpoint two; it was removed as redundant - origin fallback here is a Talos
default, not an endpoint this project maintains.</p>
<p>One asymmetry decides the whole rollout order in section 07: registry <em>auth</em> in
the machine config is documented by Talos as requiring a reboot before the CRI picks it up.
Mirror <em>endpoint</em> changes applied live in this lab (observed on Talos v1.13.4 /
containerd 2.2.4; the docs are silent on this half, so treat it as an observation, not a
guarantee). Credentials the registry consumes from a mounted secret rotate with a pod
restart. Put each credential on the side that can move.</p>
<p>One asymmetry decides section 07's rollout order. Talos's
<a href="https://docs.siderolabs.com/talos/v1.13/reference/configuration/v1alpha1/config/">v1alpha1
machine-config reference</a> states for registry auth: "changes to the registry auth will
not be picked up by the CRI containerd plugin without a reboot" - matching what this lab
saw on v1.13.4. Mirror <em>endpoint</em> changes applied live here (same versions; the
docs are silent on that half, so it stays a lab observation, not a cross-version
guarantee). Secrets the registry consumes from a mount rotate with a pod restart. Put each
credential on the side that can move.</p>
<div class="clause"><span class="k">THE HOMELAB CLAUSE</span><br>One pod, one PVC, one node
is an accepted failure domain, not a pattern. Section 09 covers what changes when the
requirements change - and what genuinely does not.</div>
<p>One pod, one PVC, one node is an accepted failure domain here, recorded as such;
section 09 covers what changes when the requirements do.</p>
<h2 id="routing"><span class="n" aria-hidden="true">03</span>Request routing and five upstreams</h2>
<p>Five registries feed one endpoint, which raises a question the first version of this page
skated past: when a node asks the mirror for <code>pause:3.10</code>, how does the mirror
know whether that means Docker Hub, GHCR, Quay, registry.k8s.io or NGC?</p>
<p>Three facts, all verified:</p>
<p>Five registries feed one endpoint: when a node asks the mirror for
<code>pause:3.10</code>, how does it know which origin that means? Three facts:</p>
<ol>
<li><strong>containerd tells the mirror where the request came from</strong> - a mirror
request carries the original registry as a query parameter:
<code>/v2/pause/manifests/3.10?ns=registry.k8s.io</code>. That is documented containerd
behaviour.</li>
<li><strong>zot v2.1.17 ignores it.</strong> There is no handling of the <code>ns</code>
parameter in the deployed version's request path, and using it for upstream selection is an
open upstream feature request (zot issue 4187) with, at last check, no maintainer response.
<li><strong>containerd names the origin</strong> - mirror requests carry it as a query
parameter: <code>/v2/pause/manifests/3.10?ns=registry.k8s.io</code> (documented).</li>
<li><strong>zot v2.1.17 ignores it</strong> - no <code>ns</code> handling exists in the
deployed request path; using it for upstream selection is open feature request zot 4187.
The URL path alone selects the local repository.</li>
<li><strong>This deployment has no per-upstream prefixes.</strong> The live sync config
gives all five upstreams <code>prefix: "**"</code> and no <code>destination</code> - one
flat namespace. On a miss, zot tries the configured upstreams in order until one has the
path. Upstream documentation's own multi-registry example instead gives each upstream a
distinct <code>destination</code> so the requested path selects the origin.</li>
<li><strong>This deployment has no per-upstream prefixes</strong> - the live sync config
gives all five upstreams <code>prefix: "**"</code> and no <code>destination</code>. On a
miss, zot tries upstreams in config order until one has the path; the docs' own
multi-registry example instead gives each a distinct <code>destination</code>.</li>
</ol>
<div class="tablewrap"><table>
<div class="tablewrap" role="region" tabindex="0" aria-label="Upstream mapping table"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Upstream map - observed configuration</caption>
<tr><th>Origin</th><th>Local namespace</th><th>Outbound auth</th><th>Sync mode</th><th>Digest preserved</th><th>On miss</th><th>Status</th></tr>
<tr><th scope="col">Origin</th><th scope="col">Local namespace</th><th scope="col">Outbound auth</th><th scope="col">Sync mode</th><th scope="col">Digest preserved</th><th scope="col">On miss</th><th scope="col">Status</th></tr>
<tr><td>docker.io</td><td rowspan="5">flat (no prefix - collision-ambiguous by construction)</td><td>Hub login (rate-limit lift)</td><td>onDemand</td><td>yes</td><td rowspan="5">upstreams tried in config order; first that resolves the path wins</td><td><span class="st ok">observed</span></td></tr>
<tr><td>ghcr.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
<tr><td>quay.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
@@ -329,23 +322,21 @@ distinct <code>destination</code> so the requested path selects the origin.</li>
<tr><td>nvcr.io</td><td>$oauthtoken + key</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
</table></div>
<p>Why has the flat namespace not bitten? Because the five origins use largely disjoint path
conventions - Hub's official images live under <code>library/</code> (the implicit prefix
behind bare names like <code>alpine</code>), NGC content sits under <code>nvidia/</code>,
registry.k8s.io has its own layout. "Largely disjoint" is a probability, not a guarantee: an
organisation name that exists on both GHCR and Quay would collide silently, and the winner
would be config order. That is an accepted risk in a lab; it is not a design.</p>
<p>The flat namespace has not bitten because the origins use largely disjoint paths -
Hub's official images live under <code>library/</code> (the implicit prefix behind bare
names like <code>alpine</code>), NGC under <code>nvidia/</code>. Largely disjoint is a
probability: an organisation existing on both GHCR and Quay would collide, and config order
would pick the winner. Accepted risk, recorded; not a design.</p>
<div class="note warn"><span class="k">PROPOSED - NOT YET DEPLOYED</span><br>The safer shape,
verified against the documentation of both halves but not yet applied here: give each
upstream a distinct <code>destination</code> prefix in zot
(<code>/docker</code>, <code>/ghcr</code>, ...), and point each Talos mirror at the
prefixed path with <code>overridePath: true</code> (which stops the automatic
<code>/v2</code> append so the prefix survives). Then the requested path itself names the
origin, and collisions become impossible rather than improbable. The migration cost: every
cached repository changes its local path, so the cache re-warms.</div>
<div class="note warn"><span class="k">PROPOSED - NOT YET DEPLOYED</span><br>The safer
shape, documented on both halves: a distinct <code>destination</code> prefix per upstream
in zot (<code>/docker</code>, <code>/ghcr</code>, ...), with each Talos mirror pointed at
the prefixed path via <code>overridePath: true</code> (suppressing the automatic
<code>/v2</code> append). The requested path then names the origin and collisions become
impossible. Migration cost: every cached repository changes local path, so the cache
re-warms.</div>
<div class="diagram"><span class="k">Diagram B &middot; request flow</span>
<div class="diagram" role="region" tabindex="0" aria-label="Request flow diagram"><span class="k">Diagram B &middot; request flow</span>
<svg viewBox="0 0 860 330" role="img" aria-labelledby="db-t db-d">
<title id="db-t">Request flow from a node through containerd to the mirror, with hit, miss, revalidation and fallback branches</title>
<desc id="db-d">A pull begins at the node's local content store. On local miss, containerd asks the mirror. A cached digest request is served from mirror storage without upstream contact. A tag request is revalidated against the upstream even when cached, on the deployed version. A storage miss triggers on-demand sync from the matching upstream. If the mirror cannot serve, containerd falls back to the origin registry.</desc>
@@ -370,8 +361,7 @@ cached repository changes its local path, so the cache re-warms.</div>
</g>
</svg>
<p class="legend">green = fully local &middot; amber = the surprising branch on v2.1.17 &middot; magenta = fallback</p>
<p class="why">The amber branch is section 04's punchline: a warm cache does not mean the
network is out of the story.</p></div>
<p class="why">The amber branch: a warm cache does not take the network out of the story.</p></div>
<h2 id="bootstrap"><span class="n" aria-hidden="true">04</span>Bootstrap, fallback and failure modes</h2>
@@ -381,43 +371,37 @@ network is out of the story.</p></div>
<div class="cap">The dependency ring: one of the orbiting parts is the mirror itself - its own image is served through the thing it is.</div>
</div></div>
<p>Every infrastructure service eventually meets the question: what do you depend on, and
what happens when you ARE the dependency? The mirror has four such loops, and each got a
different answer.</p>
<p>Four circular dependencies live in this design; each got a different answer.</p>
<h3 id="loop1">Loop 1 - the mirror's own image comes through the mirror</h3>
<p>zot runs as a container whose image lives on a registry zot mirrors. When the hosting
node boots, it asks the mirror - which is not running, because the node is booting. The
loop breaks on the documented fallback: mirror endpoints exhaust, containerd falls through
to the origin. The design decision is restraint - <em>not</em> setting
<code>skipFallback: true</code>.</p>
to the origin. Keeping <code>skipFallback</code> unset is the load-bearing choice.</p>
<div class="note"><span class="k">FIELD NOTE &middot; OBSERVED</span><br>Proven by accident:
a config change shipped with <code>preserveDigest</code> enabled but without its mandatory
partner <code>http.compat</code> - a pairing the registry refuses to start without (the
documentation is explicit; so was the crashloop). The mirror went down; the fleet quietly
fell through to upstream and nothing user-visible broke. An unplanned failover exercise,
passed. The standing rule it bought: validate the config with the registry's own
<code>verify</code> in a throwaway pod before merging, every time.</div>
<div class="note"><span class="k">FIELD NOTE &middot; LAB, 2026-06</span><br>Exercised by
accident: a config change shipped with <code>preserveDigest</code> enabled but without its
mandatory partner <code>http.compat</code> - a pairing the registry refuses to start
without. The mirror crashlooped; the fleet fell through to upstream and nothing
user-visible broke. Since then, every config change runs the registry's own
<code>verify</code> in a throwaway pod before merging.</div>
<h3 id="loop2">Loop 2 - the recovery tooling deliberately ignores the mirror</h3>
<p>The fleet's rescue tooling could pull its execution image through the mirror like
everything else. It does not: its image is cached on the operations host, outside the
cluster. A recovery tool that depends on the thing it recovers is not a recovery tool.</p>
<h3 id="loop2">Loop 2 - the recovery tooling ignores the mirror</h3>
<p>The fleet's rescue tooling does not pull through the mirror: its image is cached on the
operations host, outside the cluster, so it remains available while zot or its cluster is
down.</p>
<h3 id="loop3">Loop 3 - authentication cannot flip everywhere at once</h3>
<p>Covered properly in section 07; the loop shape: enforcing auth requires every node to
carry credentials that only apply after a reboot, so anonymous read must survive until the
last node is proven, and the proof itself must be designed not to lie.</p>
<p>Enforcing auth needs every node to carry credentials that only apply after a reboot, so
anonymous read must survive until the last node is proven - and the proof must be designed
not to lie. Section 07 covers it.</p>
<h3 id="loop4">Loop 4 - the mirror's host is also the mirror's customer</h3>
<p>The node hosting the mirror boots its own workloads through it - including the tunnel
that serves this very page. "Restart the mirror's node" therefore carries a blast radius
far beyond the mirror, and the runbook for that reboot lists every public-facing thing
riding on it and the order they return. Draw the dependency arrows for your own estate; the
ones that surprise you are the ones that will page you.</p>
<h3 id="loop4">Loop 4 - the mirror's host is also its customer</h3>
<p>The node hosting the mirror boots its own workloads through it, including the tunnel
serving this page - so restarting that node carries a blast radius beyond the mirror, and
its reboot runbook lists every public-facing dependant and the order they return.</p>
<div class="diagram"><span class="k">Diagram D &middot; two bootstrap worlds</span>
<div class="diagram" role="region" tabindex="0" aria-label="Bootstrap paths diagram"><span class="k">Diagram D &middot; two bootstrap worlds</span>
<svg viewBox="0 0 860 300" role="img" aria-labelledby="dd-t dd-d">
<title id="dd-t">Connected fail-open bootstrap versus disconnected preseeded bootstrap, as separate paths</title>
<desc id="dd-d">Left: the connected lab path - node boots, mirror miss, implicit fallback to origin, mirror comes up afterwards. Right: the disconnected path - no origin exists; the registry image and release set must be preseeded onto the host or imported from disk before anything else can start.</desc>
@@ -438,26 +422,24 @@ ones that surprise you are the ones that will page you.</p>
</g>
</svg>
<p class="legend">left = this lab, observed &middot; right = the airgap translation, a design requirement not a deployed claim</p>
<p class="why">Loop 1's trick assumes an upstream exists to fall through to. The airgap
deletes that assumption, and forces Loop 2's discipline onto the mirror itself.</p></div>
<p class="why">Loop 1 assumes an upstream to fall through to; the airgap deletes that
assumption and forces Loop 2's discipline onto the mirror itself.</p></div>
<p><strong>Fallback is a policy decision, not a universal win.</strong> This lab keeps it on:
better bootstrap survivability, availability through mirror outages, and the cost is real -
pulls can silently bypass the mirror (losing cache benefit and any future policy point) and
land on origin rate limits. A regulated or disconnected estate makes the opposite call:
<code>skipFallback: true</code>, egress restrictions, preseeded release stock, and a tested
recovery path, because silent bypass is the failure mode there, not the safety net.</p>
<p><strong>Fallback is a policy decision.</strong> This lab keeps it on for bootstrap
survivability and availability through mirror outages; the cost is that pulls can bypass
the mirror unnoticed and land on origin rate limits. A regulated or disconnected estate
makes the opposite call - <code>skipFallback: true</code>, egress restriction, preseeded
release stock, a tested recovery path - because there the bypass is the failure mode.
Failure classes differ in symptom, path and consequence:</p>
<p>And "fails fast" is not one behaviour. The failure modes differ in symptom, path and
consequence:</p>
<div class="tablewrap"><table>
<tr><th>Failure</th><th>User-visible symptom</th><th>Request path</th><th>Fallback?</th><th>Detection</th><th>Security consequence</th></tr>
<tr><td>Mirror pod down</td><td>none (pulls slower)</td><td>node -&gt; origin</td><td>yes, immediate</td><td>mirror probes red; origin egress rises</td><td>policy/audit bypass while down</td></tr>
<div class="tablewrap" role="region" tabindex="0" aria-label="Failure mode matrix"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Failure modes - symptom, path, fallback and consequence</caption>
<tr><th scope="col">Failure</th><th scope="col">User-visible symptom</th><th scope="col">Request path</th><th scope="col">Fallback?</th><th scope="col">Detection</th><th scope="col">Security consequence</th></tr>
<tr><td>Mirror pod down</td><td>public pulls may continue transparently; private images, origin limits, DNS or TLS issues can slow or fail them</td><td>node -&gt; origin</td><td>after the mirror endpoint fails; elapsed time differs by failure class (refused vs DNS vs TLS vs blackhole)</td><td>mirror probes red; origin egress rises</td><td>policy/audit bypass while down</td></tr>
<tr><td>Mirror up, sync wedged</td><td>pulls hang up to sync timeout</td><td>node -&gt; mirror (blocked)</td><td>only after timeout</td><td>manifest probe stalls while /v2/ still answers</td><td>availability, not integrity</td></tr>
<tr><td>Upstream 429</td><td>miss/tag pulls fail or crawl</td><td>mirror -&gt; origin refused</td><td>fallback hits the same limit</td><td>zot logs; retry storms amplify</td><td>none; self-inflicted DoS via retries</td></tr>
<tr><td>Upstream down, digest cached</td><td>none</td><td>mirror serves locally</td><td>not needed</td><td>-</td><td>none (verified behaviour)</td></tr>
<tr><td>Upstream down, tag cached</td><td>fails/stalls on v2.1.17</td><td>mirror insists on revalidating</td><td>origin also down</td><td>the test matrix in section 08</td><td>availability surprise - the trap of assuming "cached = offline-safe"</td></tr>
<tr><td>Upstream 429</td><td>miss/tag pulls fail or crawl</td><td>mirror -&gt; origin refused</td><td>fallback also reaches the origin and may hit a pull or abuse limit - the quota bucket depends on the node's identity and source IP, not necessarily zot's</td><td>zot logs; retry storms amplify</td><td>self-inflicted denial of service via retries</td></tr>
<tr><td>Upstream down, digest cached</td><td>none expected</td><td>mirror serves locally</td><td>not needed</td><td>-</td><td>none (source-verified short-circuit; disconnected drill still open)</td></tr>
<tr><td>Upstream down, tag cached</td><td>reported on v2.1.17: failure or waiting until timeout, with content possibly served after it</td><td>mirror revalidates the tag upstream first</td><td>origin also down</td><td>the test matrix in section 08</td><td>availability surprise for anyone assuming cached means offline-safe (upstream-reported; not yet reproduced here)</td></tr>
<tr><td>PVC full</td><td>new pulls fail, cached OK-ish</td><td>mirror 5xx on writes</td><td>yes for missing content</td><td>capacity metrics (proposed)</td><td>availability; GC pressure</td></tr>
<tr><td>Digest entry GC-evicted</td><td>silent re-fetch on next pull</td><td>mirror -&gt; origin re-sync</td><td>n/a</td><td>upstream egress for "cached" content</td><td>rate-limit exposure returns (section 06)</td></tr>
<tr><td>Node auth wrong (post-flip)</td><td>ImagePullBackOff</td><td>mirror 401; fallback unreliable on 401</td><td>unreliable</td><td>the canary gate (section 07)</td><td>the outage the gate exists to prevent</td></tr>
@@ -465,44 +447,41 @@ consequence:</p>
<h2 id="digests"><span class="n" aria-hidden="true">05</span>Digests, manifests and the supply chain</h2>
<p>Precision matters here, because half the traps on this page come from conflating these
objects:</p>
<p>Half the mistakes on this page came from conflating these objects:</p>
<ul>
<li>A <strong>tag</strong> is a mutable, human-readable pointer. Zero, one or many tags can
reference one manifest.</li>
<li>A <strong>manifest digest</strong> identifies exact manifest bytes. A multi-platform
<strong>index</strong> has its own digest; each platform manifest has another; layers and
config blobs have their own again.</li>
<li>A <strong>tag</strong> is a mutable pointer; zero or many tags can reference one
manifest.</li>
<li>A <strong>manifest digest</strong> identifies exact manifest bytes; a multi-platform
<strong>index</strong>, each platform manifest, and every layer and config blob carry their
own.</li>
<li><code>name:tag@digest</code> is client-side grammar, not wire protocol: the OCI
distribution spec takes a tag <em>or</em> a digest in the URL. Clients resolve the digest
and ignore the tag (Kubernetes documents exactly this), so digest-pinned pulls are immune
to tag moves - a fact this page once mislearned as a cache trap and later retested: the
historical zot rejection of combined references did not reproduce on v2.1.17. Traps are
perishable; retest your scars.</li>
and ignore the tag (Kubernetes documents this), so digest-pinned pulls are immune to tag
moves. This page once mislearned the combined form as a cache-miss bug; retesting on
v2.1.17 showed the historical zot rejection no longer reproduces.</li>
<li>Digest pinning provides <strong>content integrity</strong> - not provenance, not
authorisation, not availability, and not freedom from network fetches on a cold store.</li>
</ul>
<p>For a mirror, digests carry one sharp operational rule, learned here as a crashloop:
zot converts Docker-schema manifests to OCI by default, and conversion changes the digest -
silently breaking every digest pin and signature downstream. The pairing that prevents it
(<code>preserveDigest: true</code> per upstream, with <code>http.compat: ["docker2s2"]</code>)
is mandatory in both directions: the registry refuses to start with one and not the other.
This deployment runs both, on all five upstreams - observed in the live config.</p>
<p>For a mirror, digests carry one sharp operational rule: zot converts Docker-schema
manifests to OCI by default, and conversion changes the digest - breaking every digest pin
and signature downstream. The pairing that prevents it (<code>preserveDigest: true</code>
per upstream, with <code>http.compat: ["docker2s2"]</code>) is mandatory in both
directions: the registry refuses to start with one and not the other (the Loop 1 field
note is this rule being learned). This deployment runs both on all five upstreams.</p>
<p>Preserving digests does <em>not</em> mean the supply chain travels whole. OCI 1.1
referrers - signatures, SBOMs, attestations attached via the <code>subject</code> field and
served by the referrers API - are separate objects with their own discovery path. A
pull-through cache that mirrors manifests and blobs does not automatically carry them.
Nothing here verifies signatures today; if it did, disconnected verification would also
need the trust material carried locally (a public key, or a trusted-root bundle for a
private signing stack). That is stated as a boundary, not an aspiration.</p>
<div class="clause"><span class="k">THE HOMELAB CLAUSE</span><br>Production estates promote
releases by digest between registries - a tag is a suggestion, a digest is a fact - and
decide explicitly whether referrers travel with them. The lab runs the digest half of that
discipline; the referrer half is future work and labelled as such.</div>
<p>Digest preservation and artifact completeness are different properties. The docs tie
<code>preserveDigest</code>/<code>compat</code> to keeping mirrored manifest bytes and
media types - and therefore signature <em>validity</em> - aligned with upstream. Whether
signature, SBOM and attestation objects <em>arrive</em> is separate: OCI 1.1 referrers
ride the <code>subject</code> field and their own API, legacy Cosign signatures ride
tag-schema conventions, and both depend on origin support and this version's sync
behaviour. This deployment has not verified a complete referrer graph for any origin and
performs no cryptographic signature verification; disconnected verification would also
need locally held trust material. Both gaps sit in the open register. Production estates
promote releases by digest and decide per policy whether referrers travel too; this lab
runs only the digest half.</p>
<h2 id="retention"><span class="n" aria-hidden="true">06</span>Retention, GC and the digest-only trap</h2>
@@ -512,35 +491,29 @@ discipline; the referrer half is future work and labelled as such.</div>
<div class="cap">Cached bricks queued behind a jammed intake: availability problems here are quiet, not loud.</div>
</div></div>
<p>This section corrects the largest error in this page's own first edition. I wrote:
"no retention config means keep everything". That is true for tags, and dangerously false
for everything else - and "everything else" includes precisely the content this mirror
exists to hold.</p>
<p>The verified semantics, from upstream documentation and the deployed version's source:</p>
<p>This section corrects the largest error in this page's first edition, which said "no
retention config means keep everything". True for tags; false for everything else - and
"everything else" includes the content this mirror exists to hold. The semantics, from
upstream documentation and the deployed version's source:</p>
<ul>
<li>With no retention configuration, <strong>all tags are retained</strong> - and
<strong>all untagged manifests are deleted</strong> by garbage collection (unless referenced
by an index or artifact) once they are older than <code>gcDelay</code>. This deployment's
delay is the 1-hour default, with GC sweeping daily.</li>
<li>With no retention configuration, <strong>all tags are retained</strong> and
<strong>all untagged manifests are deleted</strong> by GC (unless referenced by an index
or artifact). Timing is two-stage: the manifest ages past <code>gcDelay</code> (1h here),
then deletion happens at the next sweep (<code>gcInterval</code>, 24h here) - the survival
window depends on where creation falls relative to that sweep, not a fixed one-hour fuse.</li>
<li><strong>Digest-only pull-through entries are stored as untagged manifests.</strong>
That is the exact wording of the upstream fix's problem statement: pull an image through
the mirror by digest, and the cache entry has no tag to protect it.</li>
That is the upstream fix's own problem statement: pull an image through the mirror by
digest, and the cache entry has no tag to protect it.</li>
</ul>
<div class="sayit"><span class="k">SAY IT</span>A digest-pinned fleet, pulling through a
mirror whose GC treats digest-only entries as garbage: the cache evicts exactly what the
estate is built on.</div>
<p>Combined with section 04's revalidation behaviour: <em>tag</em> pulls are cached but
contact upstream anyway; <em>digest</em> pulls serve locally but their entries are
GC-eligible between sweeps - a digest-pinned fleet caches exactly the entry class GC may
delete. On v2.1.17 this mirror is a rate-limit shield and a latency win, <strong>not yet a
disconnection shelf</strong>; the first edition implied otherwise and was wrong.</p>
<p>Put the pieces together with section 04's revalidation fact and the deployed behaviour
is this: <em>tag</em> pulls are cached but always phone upstream anyway; <em>digest</em>
pulls are served locally but their cache entries are eligible for eviction within hours.
On v2.1.17, this mirror is a rate-limit shield and a latency win - it is <strong>not yet a
disconnection shelf</strong>, and the first edition of this page was wrong to imply
otherwise.</p>
<div class="diagram"><span class="k">Diagram C &middot; what keeps an object alive</span>
<div class="diagram" role="region" tabindex="0" aria-label="Storage and retention diagram"><span class="k">Diagram C &middot; what keeps an object alive</span>
<svg viewBox="0 0 860 360" role="img" aria-labelledby="dc-t dc-d">
<title id="dc-t">The OCI object graph and which references protect content from garbage collection</title>
<desc id="dc-d">A tag points to an index; the index references platform manifests; manifests reference layers and a config blob. Referrers attach to a manifest via a subject field. A separate digest-only manifest sits with no tag pointing at it; it is eligible for garbage collection after the delay on the deployed version. Retention rules that keep tags do not protect the untagged manifest.</desc>
@@ -564,16 +537,16 @@ otherwise.</p>
<rect x="30" y="215" width="280" height="52" rx="8" fill="none" stroke="#e8b44a"/><text x="170" y="236" fill="#dfe6f0">digest-only cached manifest</text><text x="170" y="253" fill="#e8b44a">NO tag references it</text>
<rect x="380" y="215" width="220" height="52" rx="8" fill="none" stroke="#ff8fa3"/><text x="490" y="236" fill="#dfe6f0">GC after gcDelay (1h here)</text><text x="490" y="253" fill="#ff8fa3">deleteUntagged default: true</text>
<line x1="310" y1="241" x2="375" y2="241" stroke="#ff8fa3" marker-end="url(#dc-a)"/>
<text x="430" y="310" fill="#98a2b4">keepTags rules protect TAGS in their repo. Nothing on v2.1.17/18 protects the amber box;</text>
<text x="430" y="330" fill="#98a2b4">pull-aware keepUntagged ships in v2.1.19 (2026-08-04). Node-local caches can mask the eviction for days.</text>
<text x="430" y="310" fill="#98a2b4">keepTags protects TAGS in its repo. On v2.1.17/18 the amber box has one blanket lever: deleteUntagged:false,</text>
<text x="430" y="330" fill="#98a2b4">which keeps ALL untagged manifests (capacity trade). Selective pull-aware keepUntagged ships in v2.1.19.</text>
</g>
</svg>
<p class="legend">green = protected by a tag &middot; amber/red = the eviction path &middot; dashed magenta = referrers (separate lifecycle)</p>
<p class="why">The lesson: reachability, not existence, is what GC respects - and a
pull-through cache full of digest pulls is a graveyard of unreachable-by-tag objects.</p></div>
<p class="why">GC evaluates reference reachability: an untagged digest-only manifest can be
deleted even while its blobs remain present, and node-local image stores can mask the
eviction for days.</p></div>
<p>Retention policy semantics, stated precisely because two nearby systems use opposite
rules:</p>
<p>Retention policy semantics - two nearby systems use opposite matching rules:</p>
<ul>
<li>Retention policies match per repository, <strong>first match wins</strong> - order the
@@ -589,24 +562,21 @@ match. Two adjacent config blocks, two opposite precedence rules. Label which on
reasoning about.</li>
</ul>
<p><strong>Mitigations, honestly ranked:</strong> upgrade to v2.1.19+ and configure
<code>keepUntagged</code> with pull-activity rules (the designed fix); until then, widen
<code>gcDelay</code>/<code>retention.delay</code> so eviction pressure drops (the upstream
maintainer's own interim suggestion), or set <code>deleteUntagged: false</code> and accept
that the cache only grows - a capacity trade, not a free lunch. Disabling GC entirely
trades eviction for guaranteed storage exhaustion with no reclaim path; it is listed here
to be argued against.</p>
<p><strong>Mitigations, ranked:</strong> upgrade to v2.1.19+ for pull-aware
<code>keepUntagged</code> (the designed fix, schema to be validated against the actual
binary before rollout); until then, widen <code>gcDelay</code>/<code>retention.delay</code>
(the upstream maintainer's interim suggestion), or set <code>deleteUntagged: false</code>,
which protects every untagged manifest at the cost of unbounded cache growth. Disabling GC
entirely swaps eviction for storage exhaustion with no reclaim path.</p>
<div class="note warn"><span class="k">SAFE VALIDATION - PROPOSED, NOT YET RUN</span><br>
The eviction claim above is documented upstream and consistent with this config; it has not
been reproduced in this lab yet. The safe reproduction, for a <em>disposable</em> zot
instance only: record version + sanitised config; pull an image by digest through it;
confirm on the registry side that the stored manifest is untagged; shorten GC timers (in
the disposable instance only); observe the manifest before and after the sweep; then block
upstream and repeat the pull from a clean runtime store, recording whether the mirror
serves or re-fetches. Two warnings from upstream documentation: the retention verification
tool executes orphan-blob GC for real even in dry-run, and on local storage it requires the
registry stopped. Never point retention experiments at live storage.</div>
The eviction claim is documented upstream and consistent with this config; it has not been
reproduced here. The reproduction, in a <em>disposable</em> instance only: record version
and sanitised config; pull by digest; confirm the stored manifest is untagged; shorten GC
timers; observe the manifest across a sweep; then block upstream and re-pull from a clean
runtime store, recording serve vs re-fetch. Two upstream warnings: the retention
verification tool executes orphan-blob GC for real even in dry-run, and local storage
requires the registry stopped. Never point retention experiments at live storage.</div>
<h2 id="auth"><span class="n" aria-hidden="true">07</span>The authentication migration</h2>
@@ -618,22 +588,19 @@ registry stopped. Never point retention experiments at live storage.</div>
<div class="cap">The proving gate: a node counts as migrated when an authenticated pull succeeds where an anonymous one cannot.</div>
</div></div>
<p>Target state: the mirror refuses anonymous pulls. Current state: anonymous read on, with
per-node credentials staged in machine configs (inert until each node's reboot - the
documented behaviour). The migration is a sequencing problem with one trap at its centre:</p>
<p>Target state: the mirror refuses anonymous pulls. Current state: anonymous read on,
per-node credentials staged inert in machine configs until each reboot. The sequencing
problem has one central hazard: <strong>while anonymous read is on, an ordinary pull says
nothing about auth</strong> - a node whose credentials never applied is served as an
anonymous reader and false-passes the gate. The canary must live in a repository that
denies anonymous read, so success can only mean an authenticated pull.</p>
<p><strong>While anonymous read is on, an ordinary pull proves nothing about auth.</strong>
A node whose credentials never applied sends no authorisation header, gets served as an
anonymous reader, and false-passes the gate. The canary must live in a repository that
denies anonymous read, so a successful pull can only mean an authenticated pull.</p>
<p>That design is now upstream-validated rather than assumed: zot authorisation resolves
per-repository policies by <strong>longest match</strong> - the most specific path pattern
wins, and <code>**</code> is explicitly the default policy for anything unmatched. A
<code>canary/**</code> entry that grants named identities read and carries no
<code>anonymousPolicy</code> therefore denies anonymous on that path while the glob keeps
the rest of the shelf open. Maintainer guidance confirms anonymous and authenticated access
are evaluated independently. The staged policy:</p>
<p>The design is upstream-supported: authorisation resolves per-repository policies by
<strong>longest match</strong> (<code>**</code> is the default for anything unmatched), and
maintainer guidance confirms anonymous and authenticated access evaluate independently. So
a <code>canary/**</code> entry granting named identities read, with no
<code>anonymousPolicy</code>, denies anonymous on that path while the glob keeps the rest
of the shelf open. The staged policy:</p>
<pre>"accessControl": {
"repositories": {
@@ -664,23 +631,20 @@ buildah, so the caveat is noted rather than felt.</div>
<ol>
<li>Stage credentials in the node's machine config (inert; documented as requiring reboot).</li>
<li>Reboot the node at a planned window.</li>
<li><strong>Prove identity:</strong> pull the protected canary through the node's own
runtime. Success = authenticated (anonymous cannot); failure = 401, back to step 1. The
expected split, to be confirmed against observed responses when the test runs: 401 for
missing/wrong credentials, 403 for a valid identity lacking the action - upstream docs
state the 403 case explicitly only for OIDC identities, so the basic-auth 403 boundary is
listed in the open-verification register rather than asserted.</li>
<li><strong>Prove interception separately:</strong> the canary pull is a direct reference
to the mirror, so it cannot prove that <code>docker.io/...</code> references are being
routed through the mirror at all. That second proof needs an original-upstream reference
pulled on the node <em>plus</em> the mirror's request log showing it arrive - and a cold
runtime store, because node-local content will satisfy the pull without any network and
fake a pass.</li>
<li>Only when every node passes both proofs does <code>anonymousPolicy</code> come off the
glob. Rollback is the previous config commit, and it is named before the flip, not after.</li>
<li><strong>Identity check:</strong> pull the protected canary through the node's own
runtime. Success = authenticated (anonymous cannot); 401 = back to step 1. The expected
split - 401 for missing/wrong credentials, 403 for a valid identity lacking the action -
sits in the open register, since the docs state the 403 case only for OIDC identities.</li>
<li><strong>Interception check, separately:</strong> the canary is a direct mirror
reference, so it says nothing about whether <code>docker.io/...</code> references route
through the mirror. That needs an original-upstream reference pulled on a node with a cold
runtime store, correlated with the mirror's request log - node-local content would satisfy
the pull without any network and fake a pass.</li>
<li>Anonymous read comes off the glob only when every node passes both checks. Rollback is
the previous config commit, named before the flip.</li>
</ol>
<div class="diagram"><span class="k">Diagram E &middot; the migration ladder</span>
<div class="diagram" role="region" tabindex="0" aria-label="Authentication migration diagram"><span class="k">Diagram E &middot; the migration ladder</span>
<svg viewBox="0 0 860 210" role="img" aria-labelledby="de-t de-d">
<title id="de-t">The authentication migration ladder with its per-node proof gate and rollback point</title>
<desc id="de-d">Five stages: anonymous baseline, credentials staged inert, node reboot, the two-part proof - authenticated canary pull plus logged mirror interception - and the final anonymous-off flip with a named rollback commit. A failing node loops from the proof back to staging.</desc>
@@ -701,24 +665,27 @@ glob. Rollback is the previous config commit, and it is named before the flip, n
</g>
</svg>
<p class="legend">the flip is per-fleet; the proof is per-node - one unproven node under a fleet-wide flip is an outage wearing a green tick</p>
<p class="why">Why two proofs: identity and interception fail independently, and each has a
false-pass mode the other cannot detect.</p></div>
<p class="why">Identity and interception fail independently; each has a false-pass mode
the other cannot detect.</p></div>
<p>Behind the gate sit five separate trust domains, deliberately not shared: node pull
credentials (machine config, reboot-bound), CI push credentials (human-held login), the
mirror's own upstream credentials (one mounted secret, pod-restart-bound), TLS trust
(estate wildcard, standard roots), and human admin access (SSO in front of the UI). The
push credential carries this page's oldest scar: its hash and plaintext once drifted apart,
rotation became impossible, and pushes stayed dead for twelve days until the rebuilt flow
landed - three applications shipped the same day it came back. Hash and plaintext now live
side by side, rotated together by a human, and the automation identity that syncs secrets
is read-only by design so it can never half-rotate them again.</p>
<p>Five trust domains never share material: node pull credentials (machine config,
reboot-bound), the push credential (used by the operator-run build host for registry
logins, not embedded in CI configuration), the mirror's upstream credentials (one mounted
secret, pod-restart-bound), TLS trust, and human admin access (SSO in front of the UI).
The push credential carries this page's oldest incident: in July 2026 its bcrypt hash and
plaintext drifted apart in the secrets manager, rotation became impossible, and pushes
stayed dead for twelve days - three applications shipped the day the rebuilt flow landed.
Today the hash and plaintext are separate entries in the same secrets manager, rotated as
a pair by a human; the registry consumes only the derived htpasswd file, each plaintext
reaches only its consumer, and the sync identity is read-only so automation cannot
half-rotate the pair. Residual cost, stated: one secrets-manager project holds both
halves, so its compromise yields verifier and credential together - versioned,
consumer-scoped secret objects remain a possible refinement.</p>
<h2 id="observability"><span class="n" aria-hidden="true">08</span>Observability that means something</h2>
<p>"The registry is up" is the least useful sentence in this page. The worst incident in
this mirror's life: an in-flight sync wedged, every pull from that upstream hung, and the
health endpoint returned 200 throughout. The log line was almost poetic:</p>
<p>The worst incident in this mirror's life: an in-flight sync wedged, every pull from
that upstream hung, and the health endpoint returned 200 throughout:</p>
<div class="pane-k">Replay &middot; observed 2026-07-07 &middot; the stalled sync</div>
<pre class="pane">"image already demanded, waiting on channel"
@@ -729,30 +696,27 @@ $ curl -sfS --max-time 10 -o /dev/null -w '%{http_code}\n' \
-H 'Accept: application/vnd.oci.image.index.v1+json'
<span class="good">200</span> <span class="cm">&lt;- THIS is "the mirror works"</span></pre>
<p>Honesty about that incident, upgraded by reading the source: the log line is
<em>coalescing by design</em> - concurrent requests for one image join the first sync
rather than duplicating it, and since the deployed version syncs run on a detached
background context with a three-hour default timeout, surviving client disconnects.
Waiters blocking on a genuinely stalled sync until that timeout is consistent with what we
saw; the restart was a recovery action, and the root cause was never isolated. Two knobs
this config does not currently set - sync <code>maxRetries</code> (disabled by default
upstream) and a tighter <code>syncTimeout</code> - are the first things a recurrence should
change.</p>
<p>The deployed source refines that story: the log line is coalescing by design -
concurrent requests join the first sync, which runs on a detached background context with
a three-hour default timeout and survives client disconnects. Waiters blocking on a
stalled sync until that timeout matches what we saw; the restart was recovery, and the
root cause was never isolated. Two knobs this config leaves unset - sync
<code>maxRetries</code> (disabled by default upstream) and a tighter
<code>syncTimeout</code> - are the first change on recurrence.</p>
<p>What this deployment actually exposes, verified against the running version's source:
<code>/livez</code>, <code>/readyz</code> and <code>/startupz</code> exist as real health
endpoints (absent from the docs at this version - a documentation gap, not an invention),
alongside the spec's <code>/v2/</code>. The Prometheus metrics extension exists upstream
(real series names include <code>zot_http_requests_total</code>,
<code>zot_repo_storage_bytes</code>, <code>zot_repo_downloads_total</code>,
<code>zot_storage_lock_latency_seconds</code>) - and is <strong>not enabled in this
deployment</strong>. Wiring it, with scrape auth, is on the open list; no dashboard here
pretends otherwise.</p>
<p>From the running version's source: <code>/livez</code>, <code>/readyz</code> and
<code>/startupz</code> exist as health endpoints (absent from this version's docs),
alongside the spec's <code>/v2/</code>. The Prometheus metrics extension exists upstream -
series names include <code>zot_http_requests_total</code>,
<code>zot_repo_storage_bytes</code>, <code>zot_repo_downloads_total</code> and
<code>zot_storage_lock_latency_seconds</code> - and is <strong>not enabled here</strong>;
wiring it, with scrape auth, is on the open list.</p>
<p>The layered checks that would actually mean something, each answering one question:</p>
<p>The layered checks, one question each:</p>
<div class="tablewrap"><table>
<tr><th>Check</th><th>Question it answers</th><th>Status here</th></tr>
<div class="tablewrap" role="region" tabindex="0" aria-label="Layered operational checks"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Layered checks - one question each</caption>
<tr><th scope="col">Check</th><th scope="col">Question it answers</th><th scope="col">Status here</th></tr>
<tr><td><code>/livez</code> / <code>/readyz</code></td><td>is the process up / initialised</td><td><span class="st ok">available</span></td></tr>
<tr><td>authenticated manifest GET of a local-only canary</td><td>can the right identity read real content from local storage</td><td><span class="st prop">proposed</span></td></tr>
<tr><td>anonymous GET of the canary expecting 401</td><td>is the protection actually protecting</td><td><span class="st prop">proposed</span></td></tr>
@@ -762,12 +726,12 @@ pretends otherwise.</p>
<tr><td>certificate expiry, credential age</td><td>what breaks on a schedule</td><td><span class="st prop">proposed</span></td></tr>
</table></div>
<p>The test matrix for the caching claims - each cell is an experiment, not an assumption
(status: the two digest rows are documented upstream and consistent with observed
behaviour here; the disconnected rows have not been run in this lab):</p>
<p>The caching test matrix (digest rows: source-verified and consistent with behaviour
here; disconnected rows: not yet run in this lab):</p>
<div class="tablewrap"><table>
<tr><th>Scenario</th><th>Expected on v2.1.17</th><th>What proves it</th></tr>
<div class="tablewrap" role="region" tabindex="0" aria-label="Cache behaviour test matrix"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Cache behaviour - each row is an experiment</caption>
<tr><th scope="col">Scenario</th><th scope="col">Expected on v2.1.17</th><th scope="col">What proves it</th></tr>
<tr><td>cached tag, upstream reachable</td><td>served; upstream contacted anyway (revalidation)</td><td>zot log shows upstream request; latency includes round-trip</td></tr>
<tr><td>cached tag, upstream unreachable</td><td>fails or stalls until timeout, then local fallback path</td><td>pull timing vs sync timeout; error text</td></tr>
<tr><td>cached digest, upstream reachable</td><td>served locally, no upstream contact</td><td>absence of upstream request in zot logs during the pull</td></tr>
@@ -784,38 +748,33 @@ behaviour here; the disconnected rows have not been run in this lab):</p>
<div class="cap">A central curated source and site-local registries: each site holds its own shelf, so a cut WAN idles nothing.</div>
</div></div>
<p>The first edition of this section asserted what "production" does. That was the wrong
register: production chooses from patterns against requirements. The honest version:</p>
<p>The first edition of this section asserted what "production" does; production chooses
from patterns against requirements:</p>
<ul>
<li><strong>Single registry, accepted failure domain</strong> - exactly this lab's shape.
Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or to a paused
deploy window (fallback off) and that cost is accepted in writing.</li>
<li><strong>Sync-based HA</strong> - upstream documents active/standby and active/active
pairs of <em>independent</em> zot instances, each with its own storage, mirroring each
other behind a load balancer; the documented caveat is the synchronisation window between
polls, which bounds what a failover can lose.</li>
<li><strong>Scale-out clustering</strong> - upstream's other documented shape: instances
shard repositories by hash and proxy to the owner, classically over shared S3-compatible
storage. That is horizontal scale for load; the shared storage is then the availability
story, which is a different property than the HA pair above. Naming which property you are
buying is the design act.</li>
<li><strong>Edge estates</strong> - a central curated source promotes releases by digest;
each site runs a small local registry holding its release set, synced on a schedule, so a
severed WAN idles nothing at pull time. A disconnection claim is only valid when the
required content, its referrers if policy demands them, trust roots, credentials and the
registry's own bootstrap image are already local - the checklist from diagram D, applied
per site.</li>
<li><strong>Single registry, accepted failure domain</strong> - this lab's shape.
Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or a paused
deploy window (fallback off), with the cost accepted in writing.</li>
<li><strong>Sync-based HA</strong> - documented active/standby or active/active pairs of
<em>independent</em> instances, each with its own storage, mirroring each other behind a
load balancer. The caveat: the synchronisation window between polls bounds what failover
can lose.</li>
<li><strong>Scale-out clustering</strong> - instances shard repositories by hash and proxy
to the owner, classically over shared S3-compatible storage. Horizontal scale for load;
the shared storage becomes the availability story - a different property from the HA pair.</li>
<li><strong>Edge estates</strong> - a central source promotes releases by digest; each
site runs a local registry holding its release set on a schedule, so a severed WAN idles
nothing at pull time. A disconnection claim is valid only when content, required
referrers, trust roots, credentials and the registry's own bootstrap image are already
local (diagram D's right-hand column, per site).</li>
</ul>
<p>Whatever the topology: recovery time and recovery point get numbers before an incident
provides them; backups are restore-tested copies off the failure domain (array snapshots
and RAID protect against disks, not against the array, the site, or an operator error -
they are inputs to a backup strategy, not the strategy); capacity is planned against the
retention policy from section 06, because "how big does the cache get" is a policy output,
not a guess; and the circular dependencies from section 04 are drawn per site, because
every one of them exists at every scale - the only thing that changes is how expensive
they are to ignore.</p>
<p>Whatever the topology: recovery time and point objectives get numbers before an
incident supplies them; backups are restore-tested copies off the failure domain (array
snapshots and RAID protect against disk loss, not against the array, the site or operator
error); cache capacity is planned from the section 06 retention policy; and the section 04
circular dependencies get drawn per site, because each exists at every scale - only the
cost of ignoring them changes.</p>
<h2 id="verified"><span class="n" aria-hidden="true">10</span>Verified, open, and where the claims come from</h2>
@@ -824,13 +783,14 @@ they are to ignore.</p>
<div class="cap">Proof of life is one authenticated byte transfer through the real path - not a status page.</div>
</div></div>
<p>When a node powers on, this fleet's gate makes it pull one known, first-party image
through its own runtime before it counts as a member. That gate is the whole page in
miniature: prove the path, not the process.</p>
<p>When a node powers on, the fleet's admission check performs an authenticated manifest
pull through the node's own runtime before the node counts as a member: a process-health
response validates neither routing nor authorisation. Most of this page condenses into
that one command.</p>
<div class="tablewrap"><table>
<div class="tablewrap" role="region" tabindex="0" aria-label="Architecture decision table"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Architecture decisions</caption>
<tr><th>Decision</th><th>Chosen</th><th>Why</th><th>Trade-off</th><th>Rollback / alternative</th><th>Status</th></tr>
<tr><th scope="col">Decision</th><th scope="col">Chosen</th><th scope="col">Why</th><th scope="col">Trade-off</th><th scope="col">Rollback / alternative</th><th scope="col">Status</th></tr>
<tr><td>Origin fallback</td><td>ON (default kept)</td><td>bootstrap survivability, Loop 1</td><td>silent mirror bypass possible</td><td>skipFallback:true + preseed (airgap shape)</td><td><span class="st ok">observed</span></td></tr>
<tr><td>Sync mode</td><td>onDemand, all upstreams</td><td>cache follows real usage; Hub-safe</td><td>tag pulls revalidate upstream; digest entries untagged</td><td>polled sync for a curated release set</td><td><span class="st ok">observed</span></td></tr>
<tr><td>Digest preservation</td><td>preserveDigest + docker2s2, x5</td><td>digest pins + signatures survive mirroring</td><td>mandatory config pairing (crashloop scar)</td><td>none - non-negotiable for digest-pinned estates</td><td><span class="st ok">observed</span></td></tr>