mirror second-pass close-out: retention diagram precision, evidence taxonomy, sourced Talos reboot claim, qualified failure matrix, Hub pull terminology, referrer completeness split, storage semantics, secret-rotation truth, table captions + scope + keyboard regions, all-lazy plates, en-AU, editorial reduction, extended gates

2026-08-25 08:52:35 +10:00
parent 7f7035b98f
commit d443f9ad6b
6 changed files with 312 additions and 340 deletions
@@ -1,11 +1,11 @@
 # syntax=docker/dockerfile:1
-# Static teaching site: bash assembles dist/ from the split source, then a pinned
+# Static teaching site: build.sh runs the node assembler (pilot/assemble-arc.mjs) to
-# nginx-unprivileged serves it. No Node build — build.sh is plain bash + coreutils.
+# produce dist/, then a pinned nginx-unprivileged serves it. No package installs - the
 # assembler is dependency-free node.
 # ---- build stage ----------------------------------------------------------
-# Reusing the digest the site image already pins: it's a bookworm-slim base that is
+# A bookworm-slim node base, digest-pinned and typically served from the zot
-# verified and cached in the zot pull-through mirror, so this layer costs nothing
+# pull-through mirror; node runs the assembler.
 # extra. Only bash/sed/find/install are used here — node itself is incidental.
 FROM node:22-bookworm-slim@sha256:d9f850096136edbc402debdd8729579a288aac64574ada0ff4db26b6ae58b0b2 AS build
 ARG BUILD_REV=dev
 ENV BUILD_REV=$BUILD_REV
@@ -1,7 +1,7 @@
 # The Teaching Lab (learn.bztmon.com)
-A multi-page static technical learning site: scroll-driven teardowns of infrastructure the
+A static learning site documenting infrastructure I operate in my homelab - the decisions,
-author actually operates. Dark engineering-dossier aesthetic (void `#070b14`, cyan `#3fbaf5`,
+the failures, and the validation still outstanding. Dark engineering-dossier aesthetic (void `#070b14`, cyan `#3fbaf5`,
 magenta `#e879f9`), dependency-light by design, strict security posture.
 ## Pages
@@ -141,7 +141,7 @@ const OG_IMG = SITE + heroUrls["__HERO_GITOPS__"];
 // shared document shell: every page on the site gets the same head discipline
 const shell = ({ title, desc, path, ogImg, scripts, bodyHtml }) => `<!doctype html>
-<html lang="en">
+<html lang="en-AU">
 <head>
 <meta charset="utf-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
@@ -178,7 +178,7 @@ const html = shell({
 });
 // ---- The Mirror (chapter 02): fingerprint its plates + its app script -------
-const MIRROR_DESC = "A pull-through container registry taken apart: why a lab runs its own mirror, the chicken-and-egg loops inside it, and the scars that taught each rule.";
+const MIRROR_DESC = "How a zot pull-through registry behaves under Talos and containerd: upstream routing, digest preservation, garbage collection, authentication rollout, failure modes, and the lab evidence behind each decision.";
 let mirror = readFileSync(join(ROOT, "pilot/mirror.tpl.html"), "utf8");
 const mirrorPlates = {};
 for (const n of ["20", "21", "22", "23", "24", "25"]) {
@@ -186,10 +186,10 @@ for (const n of ["20", "21", "22", "23", "24", "25"]) {
  mirrorPlates[n] = url;
  const dims = jpegSize(buf);
  if (!dims) throw new Error(`mirror plate ${n}: JPEG dimensions unreadable`);
-  // plate 20 is the page's first image (near the fold) -> eager/LCP; the rest lazy.
+  // every plate sits below the fold (fact panel + contents precede the first scene),
-  const load = n === "20" ? "eager" : "lazy";
+  // so all six lazy-load; intrinsic dims come from the actual bytes.
  mirror = mirror.replace(`__M${n}__`,
-    `${url}" width="${dims.w}" height="${dims.h}" decoding="async" loading="${load}`);
+    `${url}" width="${dims.w}" height="${dims.h}" decoding="async" loading="lazy`);
 }
 const mirrorJs = readFileSync(join(ROOT, "pilot/mirror.js"));
 const mirrorJsName = `mirror.${fp(mirrorJs)}.js`;
@@ -240,6 +240,11 @@ for (const [pg, doc] of [["cluster", html], ["mirror", mirrorHtml], ["index", di
 // target the assertive phrasings and internal production jargon, not the topic words.
 const mirrorBanned = [
  [/never leaves the building/i, "retired absolute cache claim"],
  [/SAY IT/, "removed rhetorical block resurfaced"],
  [/HOMELAB CLAUSE/, "removed label resurfaced"],
  [/Nothing on v2\.1\.17\/18 protects/, "over-broad retention annotation resurfaced"],
  [/100 manifest requests per six hours/, "imprecise Hub limit phrasing"],
  [/a tag is a suggestion, a digest is a fact/i, "retired slogan"],
  [/up to twelve|twelve tries|12 attempts/i, "stale retry arithmetic (correct figure: sixteen)"],
  [/plate-level/i, "internal production note leaked to readers"],
  [/&middot; motion:/i, "art-direction caption leaked to readers"],
@@ -249,6 +254,9 @@ const mirrorBanned = [
 for (const [re, msg] of mirrorBanned) if (re.test(mirrorHtml)) fail.push("mirror: " + msg);
 const mirrorRequired = [
  ["zotregistry.dev", "primary zot references missing"],
  ["docs.docker.com/docker-hub/usage/pulls", "Docker Hub pulls reference missing"],
  ["v1alpha1", "Talos auth-reboot source link missing"],
  ["deleteUntagged", "blanket untagged lever discussion missing"],
  ["keepUntagged", "version-gated retention discussion missing"],
  ["/livez", "real health endpoints missing"],
  ['id="refs"', "references section missing"],
@@ -257,6 +265,9 @@ const mirrorRequired = [
 ];
 for (const [needle, msg] of mirrorRequired) if (!mirrorHtml.includes(needle)) fail.push("mirror: " + msg);
 must((mirrorHtml.match(/width="1600" height="872"/g) || []).length === 6, "mirror: expected six correctly-dimensioned plates");
 must((mirrorHtml.match(/loading="lazy"/g) || []).length === 6, "mirror: expected all six plates lazy-loaded");
 must((mirrorHtml.match(/<caption/g) || []).length === 5, "mirror: expected five table captions");
 must(!/<th>(?!<)/.test(mirrorHtml.replace(/<th scope="col">/g, "")), "mirror: th without scope=col");
 must(!/100vw/.test(mirrorHtml), "mirror: viewport-width unit reintroduces scrollbar overflow");
 must(!mirrorHtml.includes("gsap"), "mirror: unexpected animation library reference");
@@ -61,13 +61,13 @@ footer a{color:var(--cyan)}
  </a>
  <a class="card" href="/mirror">
-    <span class="thumb"><img src="__THUMB_MIRROR__" alt="" width="1600" height="873" loading="eager" decoding="async"></span>
+    <span class="thumb"><img src="__THUMB_MIRROR__" alt="" width="1600" height="872" loading="eager" decoding="async"></span>
    <span class="meta">
      <span class="k">Chapter 02 &middot; Dossier 005</span>
      <h2>The Mirror <span class="arrow">&rarr;</span></h2>
-      <p class="outcome"><b>Outcome</b> &mdash; a pull-through container registry taken apart: why a
+      <p class="outcome"><b>Outcome</b> &mdash; how a zot pull-through registry behaves under Talos and
-      lab runs its own mirror, the four chicken-and-egg loops hiding inside it, and the scars that
+      containerd: upstream routing, digest preservation, garbage collection, authentication
-      taught each rule - told with the receipts.</p>
+      rollout and failure modes, with the lab evidence behind each decision.</p>
    </span>
  </a>
@@ -29,6 +29,7 @@
      if (hit) visible[hit[0]] = e.isIntersecting;
    });
    if (visible.s23) startPulse(); // (re)arm the breather when its scene returns
    onScroll(); // anchor jumps and observer-only transitions update without a scroll event
  }, { rootMargin: "10% 0px" });
  watched.forEach(function (w) { io.observe(w[1]); });
@@ -37,9 +37,7 @@
  .scene img{width:100%;height:auto;display:block;border-radius:.4rem}
  .scene .cap{font-family:var(--mono);font-size:.74rem;color:var(--ink-dim);
    letter-spacing:.1em;text-align:center;margin-top:.7rem;padding:0 1rem}
-  .sayit{border-left:3px solid var(--magenta);margin:2rem 0;padding:.4rem 0 .4rem 1.1rem;
+  [tabindex="0"]:focus-visible{outline:2px solid var(--cyan);outline-offset:3px}
    font-size:1.26rem;font-weight:650;line-height:1.35;color:#fff}
  .sayit .k{display:block;font-family:var(--mono);font-size:.7rem;letter-spacing:.2em;color:var(--magenta);margin-bottom:.3rem;font-weight:500}
  .note{background:var(--card);border:1px solid var(--line);border-left:3px solid var(--cyan);
    border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
  .note .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan)}
@@ -114,11 +112,10 @@
  <p class="backlink"><a href="/">&larr; The Teaching Lab</a></p>
  <p class="eyebrow">Dossier / 005 &middot; operations</p>
  <h1>The Mirror</h1>
-  <p class="outcome"><b>Outcome</b> &mdash; a pull-through OCI registry taken apart: why this
+  <p class="outcome"><b>Outcome</b> &mdash; a pull-through OCI registry taken apart: what
-  lab runs its own mirror, what the deployed version actually guarantees, the retention trap
+  the deployed version guarantees, the retention behaviour that evicts digest-pinned
-  that quietly evicts digest-pinned content, and how each lesson translates into production
+  content, and how each lesson translates to production. Claims carry evidence labels
-  platform engineering. Claims are labelled: observed here, documented upstream, or not yet
+  throughout.</p>
  verified.</p>
 </header>
 <div class="facts">
@@ -129,9 +126,16 @@
    <dt>Storage</dt><dd>50Gi local PVC; dedupe on, GC on (1h delay / 24h interval); no retention config</dd>
    <dt>Auth state</dt><dd>anonymous read enabled; per-node pull credentials staged, not yet enforced</dd>
    <dt>Fallback</dt><dd>origin fallback ON (platform default; deliberately not disabled)</dd>
-    <dt>Validated</dt><dd>2026-08-25 (live config read + upstream doc/source verification)</dd>
+    <dt>Reviewed</dt><dd>2026-08-25: live cluster configuration read; upstream docs and v2.1.17 source checked; live runtime tests listed separately in the open register</dd>
-    <dt>Open items</dt><dd>digest-entry retention (needs v2.1.19+), auth canary negative test, upstream namespace prefixes, disconnected-serve drill</dd>
+    <dt>Open items</dt><dd>pull-aware digest-entry retention (needs v2.1.19+; blanket alternatives are capacity-heavy), auth canary negative test, upstream namespace prefixes, disconnected-serve drill</dd>
  </dl>
  <p style="margin:.9rem 0 0;font-size:.8rem;color:var(--ink-dim);font-family:var(--mono)">
  Evidence labels used below: <span class="st ok">lab</span> reproduced here with a dated receipt &middot;
  <span class="st ok">config</span> present in the inspected config &middot;
  <span class="st ok">source</span> established from the deployed version's source &middot;
  <span class="st ok">docs</span> stated in version-appropriate documentation &middot;
  <span class="st open">reported</span> described in an unresolved upstream issue &middot;
  <span class="st prop">proposed</span> not deployed &middot; <span class="st open">open</span> not yet run</p>
 </div>
 <nav class="toc" aria-labelledby="toc-h">
@@ -157,24 +161,27 @@
  <div class="cap">The mirror as a machine: upstream intakes, one cache shelf, one serving nozzle.</div>
 </div></div>
-<p>Nobody sets out to run a registry. You inherit the need the day a CI build stalls for no
+<p>I added zot after a CI pull hit Docker Hub's anonymous rate limit and the retry wrapper
-reason you can see.</p>
+hid the 429. At the time of the incident, this site's build-stage pull was anonymous.</p>
-<p>This site's own build pulls one base image from Docker Hub, anonymously. One evening the
+<p>Docker Hub does not go silent when it limits you. It answers with HTTP 429 and the error
-build sat there - no visible error, no progress. The accurate version of that story: Docker
+code <code>toomanyrequests</code>. Under the
-Hub does not go silent when it limits you. It answers with an explicit HTTP 429 and the error
+<a href="https://docs.docker.com/docker-hub/usage/pulls/">current published limits</a>,
-code <code>toomanyrequests</code>; under the current published limits, anonymous pulls get
+unauthenticated users get 100 pulls per six hours per IPv4 address (or IPv6 /64); a
-100 manifest requests per six hours per IPv4 address, and the accounting is per manifest
+single-platform image counts as one pull, a multi-architecture image counts once per
-<code>GET</code>. The <em>silence</em> was manufactured on my side, by retries swallowing
+architecture pulled, version checks do not count, and a <code>HEAD</code> request can read
-that answer.</p>
+the rate-limit headers without consuming a pull. A separate abuse limiter covers all
 request types with its own 429 form. The <em>silence</em> in this incident was
 manufactured on my side, by retries swallowing the answer.</p>
-<p>The swallower, recovered from the build script's git history, was a double retry stack: an
+<p>The swallower, recovered from the build script's git history, was a double retry stack:
-outer shell wrapper allowing four attempts, wrapped around <code>buildah build --retry 3</code>.
+an outer shell wrapper allowing four attempts, wrapped around
-Buildah's flag counts <em>retries</em>, not attempts - one initial try plus up to three
+<code>buildah build --retry 3</code>. Buildah's flag counts <em>retries</em> - one initial
-retries, and it applies to registry push/pull operations. Four outer attempts, each
+try plus up to three more, applying to registry push/pull operations. Four outer attempts,
-containing up to four inner attempts: <strong>up to sixteen requests</strong> for one
+each containing up to four inner attempts: <strong>up to sixteen requests</strong> for one
-consistently failing pull, each one billed against the same rate limit it was trying to
+repeatedly failing registry operation (a build with several pulls can produce more), each
-outlast. Retry amplification turns a throttle into an outage and hides the evidence.</p>
+billed against the limit it was trying to outlast. Retry amplification turns a throttle
 into an outage and hides the evidence while doing it.</p>
 <div class="pane-k">Receipt &middot; the retry stack, as committed to the repo</div>
 <pre class="pane">retry() {
@@ -188,30 +195,21 @@ retry buildah build --retry 3 --retry-delay 5s ...
 <span class="cm"># --retry 3 = one attempt + three retries; the wrapper multiplies that by four.</span>
 <span class="cm"># A 429 answered every one of them, and the backoffs read as a hang.</span></pre>
-<p>The immediate fix was to pin the base image by digest - which stops tag re-resolution, but
+<p>Pinning the base image by digest stopped the tag re-resolution but not the round-trip:
-does not remove the network round-trip: an ephemeral runner with a cold local store still
+an ephemeral runner with a cold store still fetches the manifest behind that digest every
-fetches the manifest behind that digest every build. The durable answer is structural:</p>
+build. The repeated origin pulls justified a shared cache. This page takes it apart,
-
+including the parts that turned out not to work the way I first believed.</p>
 <div class="sayit"><span class="k">SAY IT</span>Why does a fleet of machines ask the public
 internet for the same bytes, hundreds of times, forever?</div>
 <p>A production platform answers with a mirror: one registry that fetches once and serves the
 estate. The lab built one. This page takes it apart - including the parts that turned out
 not to work the way I first believed.</p>
 <div class="note"><span class="k">FIELD NOTE</span><br>The mirror did not fix the red X in
 CI. That red was a separate upstream bug in the CI system's log-finalise step - cosmetic,
-tolerated, documented. Two problems, one symptom. Diagnose them separately or you will fix
+tolerated, documented. Two problems, one symptom; diagnose them separately.</div>
 the wrong one and declare victory.</div>
 <h2 id="anatomy"><span class="n" aria-hidden="true">02</span>What the mirror actually does</h2>
-<p>The mirror is zot: a single OCI registry in its own namespace, one 50Gi cache volume,
+<p>The mirror is zot: one registry, one 50Gi cache volume, digest-pinned like everything
-running the same digest-pinned deployment discipline as everything it serves. It fronts five
+it serves. Five public registries front it in <strong>on-demand pull-through</strong> mode -
-public registries in <strong>on-demand pull-through</strong> mode: a miss fetches from
+a miss fetches and caches, a hit serves from the shelf. zot also offers polled mirroring
-upstream and caches; a hit serves from the shelf. zot also supports polled mirroring
+and pre-seeding; Docker Hub is on-demand only, per its documentation.</p>
 (periodic full sync of matching content) and pre-seeding; Docker Hub is on-demand only -
 upstream documentation is explicit that polled mirroring should not be pointed at Hub.</p>
 <p>The storage block, from the live config:</p>
@@ -224,12 +222,16 @@ upstream documentation is explicit that polled mirroring should not be pointed a
  "gcInterval": "24h"
 }</pre>
-<p><code>commit</code> fsyncs writes before acknowledging - crash safety. <code>dedupe</code>
+<p><code>commit</code> asks zot to commit writes to disk immediately instead of relying on
-hard-links identical blobs, which pays for itself when five upstreams ship the same base
+buffered flushing; it narrows the buffered-write window, while end-to-end power-loss
-layers under different names. And <code>gc</code> with its two timers looks innocuous here;
+durability still depends on the filesystem, volume and disk. <code>dedupe</code> uses hard
-section 06 is about why it is the most consequential block on this page.</p>
+links on local filesystem storage (remote backends implement it differently), saving
 capacity when five upstreams ship the same base layers under different names - at the cost
 of a startup reconciliation pass that matters operationally when toggled on existing data.
 And <code>gc</code> with its two timers looks innocuous here; section 06 is about why it is
 the most consequential block on this page.</p>
-<div class="diagram"><span class="k">Diagram A &middot; system context</span>
+<div class="diagram" role="region" tabindex="0" aria-label="System context diagram"><span class="k">Diagram A &middot; system context</span>
 <svg viewBox="0 0 860 400" role="img" aria-labelledby="da-t da-d">
  <title id="da-t">System context: consumers, the mirror, its storage, five upstreams, and the trust boundaries between them</title>
  <desc id="da-d">CI and Kubernetes nodes pull from the zot mirror over the LAN. The mirror stores content on a 50Gi volume and syncs on demand from five upstream registries across the internet boundary. Anonymous read is currently allowed inbound; outbound upstream credentials live in one mounted secret. The origin-fallback path from nodes directly to upstreams is shown dashed.</desc>
@@ -261,10 +263,9 @@ section 06 is about why it is the most consequential block on this page.</p>
  </g>
 </svg>
 <p class="legend">solid = normal pull path &middot; dashed cyan = on-demand sync on miss &middot; dashed magenta = origin fallback</p>
-<p class="why">The decision this explains: one cache serves two consumer classes, and the
+<p class="why">One cache, two consumer classes; the two credential domains never mix.</p></div>
 two credential domains (nodes-to-mirror, mirror-to-upstreams) never mix.</p></div>
-<p>Consumers reach the mirror through the platform's machine-level registry config:</p>
+<p>Consumers point at it via the platform's machine-level registry config:</p>
 <pre>machine:
  registries:
@@ -276,52 +277,44 @@ two credential domains (nodes-to-mirror, mirror-to-upstreams) never mix.</p></di
        endpoints:
          - https://zot.bztmon.org</pre>
-<p>The mirror is the only listed endpoint, and that is still not a hard dependency, because
+<p>The mirror is the only listed endpoint, yet not a hard dependency: Talos tries
-Talos documents an implicit final fallback: endpoints are tried in order, "and by default
+endpoints in order "and by default the last implicit endpoint is the original upstream
-the last implicit endpoint is the original upstream registry", unless
+registry", unless <code>skipFallback: true</code>. An early version listed the upstream
-<code>skipFallback: true</code> says otherwise. The lab's first version listed the upstream
+explicitly as endpoint two; it was removed as redundant - origin fallback here is a Talos
-as an explicit second endpoint; the refinement that removed it came from reading the
+default, not an endpoint this project maintains.</p>
 behaviour properly - the default already guaranteed it. Know which of your safety nets you
 built, and which are defaults you merely have not broken.</p>
-<p>One asymmetry decides the whole rollout order in section 07: registry <em>auth</em> in
+<p>One asymmetry decides section 07's rollout order. Talos's
-the machine config is documented by Talos as requiring a reboot before the CRI picks it up.
+<a href="https://docs.siderolabs.com/talos/v1.13/reference/configuration/v1alpha1/config/">v1alpha1
-Mirror <em>endpoint</em> changes applied live in this lab (observed on Talos v1.13.4 /
+machine-config reference</a> states for registry auth: "changes to the registry auth will
-containerd 2.2.4; the docs are silent on this half, so treat it as an observation, not a
+not be picked up by the CRI containerd plugin without a reboot" - matching what this lab
-guarantee). Credentials the registry consumes from a mounted secret rotate with a pod
+saw on v1.13.4. Mirror <em>endpoint</em> changes applied live here (same versions; the
-restart. Put each credential on the side that can move.</p>
+docs are silent on that half, so it stays a lab observation, not a cross-version
 guarantee). Secrets the registry consumes from a mount rotate with a pod restart. Put each
 credential on the side that can move.</p>
-<div class="clause"><span class="k">THE HOMELAB CLAUSE</span><br>One pod, one PVC, one node
+<p>One pod, one PVC, one node is an accepted failure domain here, recorded as such;
-is an accepted failure domain, not a pattern. Section 09 covers what changes when the
+section 09 covers what changes when the requirements do.</p>
 requirements change - and what genuinely does not.</div>
 <h2 id="routing"><span class="n" aria-hidden="true">03</span>Request routing and five upstreams</h2>
-<p>Five registries feed one endpoint, which raises a question the first version of this page
+<p>Five registries feed one endpoint: when a node asks the mirror for
-skated past: when a node asks the mirror for <code>pause:3.10</code>, how does the mirror
+<code>pause:3.10</code>, how does it know which origin that means? Three facts:</p>
 know whether that means Docker Hub, GHCR, Quay, registry.k8s.io or NGC?</p>
 <p>Three facts, all verified:</p>
 <ol>
-<li><strong>containerd tells the mirror where the request came from</strong> - a mirror
+<li><strong>containerd names the origin</strong> - mirror requests carry it as a query
-request carries the original registry as a query parameter:
+parameter: <code>/v2/pause/manifests/3.10?ns=registry.k8s.io</code> (documented).</li>
-<code>/v2/pause/manifests/3.10?ns=registry.k8s.io</code>. That is documented containerd
+<li><strong>zot v2.1.17 ignores it</strong> - no <code>ns</code> handling exists in the
-behaviour.</li>
+deployed request path; using it for upstream selection is open feature request zot 4187.
 <li><strong>zot v2.1.17 ignores it.</strong> There is no handling of the <code>ns</code>
 parameter in the deployed version's request path, and using it for upstream selection is an
 open upstream feature request (zot issue 4187) with, at last check, no maintainer response.
 The URL path alone selects the local repository.</li>
-<li><strong>This deployment has no per-upstream prefixes.</strong> The live sync config
+<li><strong>This deployment has no per-upstream prefixes</strong> - the live sync config
-gives all five upstreams <code>prefix: "**"</code> and no <code>destination</code> - one
+gives all five upstreams <code>prefix: "**"</code> and no <code>destination</code>. On a
-flat namespace. On a miss, zot tries the configured upstreams in order until one has the
+miss, zot tries upstreams in config order until one has the path; the docs' own
-path. Upstream documentation's own multi-registry example instead gives each upstream a
+multi-registry example instead gives each a distinct <code>destination</code>.</li>
 distinct <code>destination</code> so the requested path selects the origin.</li>
 </ol>
-<div class="tablewrap"><table>
+<div class="tablewrap" role="region" tabindex="0" aria-label="Upstream mapping table"><table>
 <caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Upstream map - observed configuration</caption>
-<tr><th>Origin</th><th>Local namespace</th><th>Outbound auth</th><th>Sync mode</th><th>Digest preserved</th><th>On miss</th><th>Status</th></tr>
+<tr><th scope="col">Origin</th><th scope="col">Local namespace</th><th scope="col">Outbound auth</th><th scope="col">Sync mode</th><th scope="col">Digest preserved</th><th scope="col">On miss</th><th scope="col">Status</th></tr>
 <tr><td>docker.io</td><td rowspan="5">flat (no prefix - collision-ambiguous by construction)</td><td>Hub login (rate-limit lift)</td><td>onDemand</td><td>yes</td><td rowspan="5">upstreams tried in config order; first that resolves the path wins</td><td><span class="st ok">observed</span></td></tr>
 <tr><td>ghcr.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
 <tr><td>quay.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
@@ -329,23 +322,21 @@ distinct <code>destination</code> so the requested path selects the origin.</li>
 <tr><td>nvcr.io</td><td>$oauthtoken + key</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
 </table></div>
-<p>Why has the flat namespace not bitten? Because the five origins use largely disjoint path
+<p>The flat namespace has not bitten because the origins use largely disjoint paths -
-conventions - Hub's official images live under <code>library/</code> (the implicit prefix
+Hub's official images live under <code>library/</code> (the implicit prefix behind bare
-behind bare names like <code>alpine</code>), NGC content sits under <code>nvidia/</code>,
+names like <code>alpine</code>), NGC under <code>nvidia/</code>. Largely disjoint is a
-registry.k8s.io has its own layout. "Largely disjoint" is a probability, not a guarantee: an
+probability: an organisation existing on both GHCR and Quay would collide, and config order
-organisation name that exists on both GHCR and Quay would collide silently, and the winner
+would pick the winner. Accepted risk, recorded; not a design.</p>
 would be config order. That is an accepted risk in a lab; it is not a design.</p>
-<div class="note warn"><span class="k">PROPOSED - NOT YET DEPLOYED</span><br>The safer shape,
+<div class="note warn"><span class="k">PROPOSED - NOT YET DEPLOYED</span><br>The safer
-verified against the documentation of both halves but not yet applied here: give each
+shape, documented on both halves: a distinct <code>destination</code> prefix per upstream
-upstream a distinct <code>destination</code> prefix in zot
+in zot (<code>/docker</code>, <code>/ghcr</code>, ...), with each Talos mirror pointed at
-(<code>/docker</code>, <code>/ghcr</code>, ...), and point each Talos mirror at the
+the prefixed path via <code>overridePath: true</code> (suppressing the automatic
-prefixed path with <code>overridePath: true</code> (which stops the automatic
+<code>/v2</code> append). The requested path then names the origin and collisions become
-<code>/v2</code> append so the prefix survives). Then the requested path itself names the
+impossible. Migration cost: every cached repository changes local path, so the cache
-origin, and collisions become impossible rather than improbable. The migration cost: every
+re-warms.</div>
 cached repository changes its local path, so the cache re-warms.</div>
-<div class="diagram"><span class="k">Diagram B &middot; request flow</span>
+<div class="diagram" role="region" tabindex="0" aria-label="Request flow diagram"><span class="k">Diagram B &middot; request flow</span>
 <svg viewBox="0 0 860 330" role="img" aria-labelledby="db-t db-d">
  <title id="db-t">Request flow from a node through containerd to the mirror, with hit, miss, revalidation and fallback branches</title>
  <desc id="db-d">A pull begins at the node's local content store. On local miss, containerd asks the mirror. A cached digest request is served from mirror storage without upstream contact. A tag request is revalidated against the upstream even when cached, on the deployed version. A storage miss triggers on-demand sync from the matching upstream. If the mirror cannot serve, containerd falls back to the origin registry.</desc>
@@ -370,8 +361,7 @@ cached repository changes its local path, so the cache re-warms.</div>
  </g>
 </svg>
 <p class="legend">green = fully local &middot; amber = the surprising branch on v2.1.17 &middot; magenta = fallback</p>
-<p class="why">The amber branch is section 04's punchline: a warm cache does not mean the
+<p class="why">The amber branch: a warm cache does not take the network out of the story.</p></div>
 network is out of the story.</p></div>
 <h2 id="bootstrap"><span class="n" aria-hidden="true">04</span>Bootstrap, fallback and failure modes</h2>
@@ -381,43 +371,37 @@ network is out of the story.</p></div>
  <div class="cap">The dependency ring: one of the orbiting parts is the mirror itself - its own image is served through the thing it is.</div>
 </div></div>
-<p>Every infrastructure service eventually meets the question: what do you depend on, and
+<p>Four circular dependencies live in this design; each got a different answer.</p>
 what happens when you ARE the dependency? The mirror has four such loops, and each got a
 different answer.</p>
 <h3 id="loop1">Loop 1 - the mirror's own image comes through the mirror</h3>
 <p>zot runs as a container whose image lives on a registry zot mirrors. When the hosting
 node boots, it asks the mirror - which is not running, because the node is booting. The
 loop breaks on the documented fallback: mirror endpoints exhaust, containerd falls through
-to the origin. The design decision is restraint - <em>not</em> setting
+to the origin. Keeping <code>skipFallback</code> unset is the load-bearing choice.</p>
 <code>skipFallback: true</code>.</p>
-<div class="note"><span class="k">FIELD NOTE &middot; OBSERVED</span><br>Proven by accident:
+<div class="note"><span class="k">FIELD NOTE &middot; LAB, 2026-06</span><br>Exercised by
-a config change shipped with <code>preserveDigest</code> enabled but without its mandatory
+accident: a config change shipped with <code>preserveDigest</code> enabled but without its
-partner <code>http.compat</code> - a pairing the registry refuses to start without (the
+mandatory partner <code>http.compat</code> - a pairing the registry refuses to start
-documentation is explicit; so was the crashloop). The mirror went down; the fleet quietly
+without. The mirror crashlooped; the fleet fell through to upstream and nothing
-fell through to upstream and nothing user-visible broke. An unplanned failover exercise,
+user-visible broke. Since then, every config change runs the registry's own
-passed. The standing rule it bought: validate the config with the registry's own
+<code>verify</code> in a throwaway pod before merging.</div>
 <code>verify</code> in a throwaway pod before merging, every time.</div>
-<h3 id="loop2">Loop 2 - the recovery tooling deliberately ignores the mirror</h3>
+<h3 id="loop2">Loop 2 - the recovery tooling ignores the mirror</h3>
-<p>The fleet's rescue tooling could pull its execution image through the mirror like
+<p>The fleet's rescue tooling does not pull through the mirror: its image is cached on the
-everything else. It does not: its image is cached on the operations host, outside the
+operations host, outside the cluster, so it remains available while zot or its cluster is
-cluster. A recovery tool that depends on the thing it recovers is not a recovery tool.</p>
+down.</p>
 <h3 id="loop3">Loop 3 - authentication cannot flip everywhere at once</h3>
-<p>Covered properly in section 07; the loop shape: enforcing auth requires every node to
+<p>Enforcing auth needs every node to carry credentials that only apply after a reboot, so
-carry credentials that only apply after a reboot, so anonymous read must survive until the
+anonymous read must survive until the last node is proven - and the proof must be designed
-last node is proven, and the proof itself must be designed not to lie.</p>
+not to lie. Section 07 covers it.</p>
-<h3 id="loop4">Loop 4 - the mirror's host is also the mirror's customer</h3>
+<h3 id="loop4">Loop 4 - the mirror's host is also its customer</h3>
-<p>The node hosting the mirror boots its own workloads through it - including the tunnel
+<p>The node hosting the mirror boots its own workloads through it, including the tunnel
-that serves this very page. "Restart the mirror's node" therefore carries a blast radius
+serving this page - so restarting that node carries a blast radius beyond the mirror, and
-far beyond the mirror, and the runbook for that reboot lists every public-facing thing
+its reboot runbook lists every public-facing dependant and the order they return.</p>
 riding on it and the order they return. Draw the dependency arrows for your own estate; the
 ones that surprise you are the ones that will page you.</p>
-<div class="diagram"><span class="k">Diagram D &middot; two bootstrap worlds</span>
+<div class="diagram" role="region" tabindex="0" aria-label="Bootstrap paths diagram"><span class="k">Diagram D &middot; two bootstrap worlds</span>
 <svg viewBox="0 0 860 300" role="img" aria-labelledby="dd-t dd-d">
  <title id="dd-t">Connected fail-open bootstrap versus disconnected preseeded bootstrap, as separate paths</title>
  <desc id="dd-d">Left: the connected lab path - node boots, mirror miss, implicit fallback to origin, mirror comes up afterwards. Right: the disconnected path - no origin exists; the registry image and release set must be preseeded onto the host or imported from disk before anything else can start.</desc>
@@ -438,26 +422,24 @@ ones that surprise you are the ones that will page you.</p>
  </g>
 </svg>
 <p class="legend">left = this lab, observed &middot; right = the airgap translation, a design requirement not a deployed claim</p>
-<p class="why">Loop 1's trick assumes an upstream exists to fall through to. The airgap
+<p class="why">Loop 1 assumes an upstream to fall through to; the airgap deletes that
-deletes that assumption, and forces Loop 2's discipline onto the mirror itself.</p></div>
+assumption and forces Loop 2's discipline onto the mirror itself.</p></div>
-<p><strong>Fallback is a policy decision, not a universal win.</strong> This lab keeps it on:
+<p><strong>Fallback is a policy decision.</strong> This lab keeps it on for bootstrap
-better bootstrap survivability, availability through mirror outages, and the cost is real -
+survivability and availability through mirror outages; the cost is that pulls can bypass
-pulls can silently bypass the mirror (losing cache benefit and any future policy point) and
+the mirror unnoticed and land on origin rate limits. A regulated or disconnected estate
-land on origin rate limits. A regulated or disconnected estate makes the opposite call:
+makes the opposite call - <code>skipFallback: true</code>, egress restriction, preseeded
-<code>skipFallback: true</code>, egress restrictions, preseeded release stock, and a tested
+release stock, a tested recovery path - because there the bypass is the failure mode.
-recovery path, because silent bypass is the failure mode there, not the safety net.</p>
+Failure classes differ in symptom, path and consequence:</p>
-<p>And "fails fast" is not one behaviour. The failure modes differ in symptom, path and
+<div class="tablewrap" role="region" tabindex="0" aria-label="Failure mode matrix"><table>
-consequence:</p>
+<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Failure modes - symptom, path, fallback and consequence</caption>
-
+<tr><th scope="col">Failure</th><th scope="col">User-visible symptom</th><th scope="col">Request path</th><th scope="col">Fallback?</th><th scope="col">Detection</th><th scope="col">Security consequence</th></tr>
-<div class="tablewrap"><table>
+<tr><td>Mirror pod down</td><td>public pulls may continue transparently; private images, origin limits, DNS or TLS issues can slow or fail them</td><td>node -&gt; origin</td><td>after the mirror endpoint fails; elapsed time differs by failure class (refused vs DNS vs TLS vs blackhole)</td><td>mirror probes red; origin egress rises</td><td>policy/audit bypass while down</td></tr>
 <tr><th>Failure</th><th>User-visible symptom</th><th>Request path</th><th>Fallback?</th><th>Detection</th><th>Security consequence</th></tr>
 <tr><td>Mirror pod down</td><td>none (pulls slower)</td><td>node -&gt; origin</td><td>yes, immediate</td><td>mirror probes red; origin egress rises</td><td>policy/audit bypass while down</td></tr>
 <tr><td>Mirror up, sync wedged</td><td>pulls hang up to sync timeout</td><td>node -&gt; mirror (blocked)</td><td>only after timeout</td><td>manifest probe stalls while /v2/ still answers</td><td>availability, not integrity</td></tr>
-<tr><td>Upstream 429</td><td>miss/tag pulls fail or crawl</td><td>mirror -&gt; origin refused</td><td>fallback hits the same limit</td><td>zot logs; retry storms amplify</td><td>none; self-inflicted DoS via retries</td></tr>
+<tr><td>Upstream 429</td><td>miss/tag pulls fail or crawl</td><td>mirror -&gt; origin refused</td><td>fallback also reaches the origin and may hit a pull or abuse limit - the quota bucket depends on the node's identity and source IP, not necessarily zot's</td><td>zot logs; retry storms amplify</td><td>self-inflicted denial of service via retries</td></tr>
-<tr><td>Upstream down, digest cached</td><td>none</td><td>mirror serves locally</td><td>not needed</td><td>-</td><td>none (verified behaviour)</td></tr>
+<tr><td>Upstream down, digest cached</td><td>none expected</td><td>mirror serves locally</td><td>not needed</td><td>-</td><td>none (source-verified short-circuit; disconnected drill still open)</td></tr>
-<tr><td>Upstream down, tag cached</td><td>fails/stalls on v2.1.17</td><td>mirror insists on revalidating</td><td>origin also down</td><td>the test matrix in section 08</td><td>availability surprise - the trap of assuming "cached = offline-safe"</td></tr>
+<tr><td>Upstream down, tag cached</td><td>reported on v2.1.17: failure or waiting until timeout, with content possibly served after it</td><td>mirror revalidates the tag upstream first</td><td>origin also down</td><td>the test matrix in section 08</td><td>availability surprise for anyone assuming cached means offline-safe (upstream-reported; not yet reproduced here)</td></tr>
 <tr><td>PVC full</td><td>new pulls fail, cached OK-ish</td><td>mirror 5xx on writes</td><td>yes for missing content</td><td>capacity metrics (proposed)</td><td>availability; GC pressure</td></tr>
 <tr><td>Digest entry GC-evicted</td><td>silent re-fetch on next pull</td><td>mirror -&gt; origin re-sync</td><td>n/a</td><td>upstream egress for "cached" content</td><td>rate-limit exposure returns (section 06)</td></tr>
 <tr><td>Node auth wrong (post-flip)</td><td>ImagePullBackOff</td><td>mirror 401; fallback unreliable on 401</td><td>unreliable</td><td>the canary gate (section 07)</td><td>the outage the gate exists to prevent</td></tr>
@@ -465,44 +447,41 @@ consequence:</p>
 <h2 id="digests"><span class="n" aria-hidden="true">05</span>Digests, manifests and the supply chain</h2>
-<p>Precision matters here, because half the traps on this page come from conflating these
+<p>Half the mistakes on this page came from conflating these objects:</p>
 objects:</p>
 <ul>
-<li>A <strong>tag</strong> is a mutable, human-readable pointer. Zero, one or many tags can
+<li>A <strong>tag</strong> is a mutable pointer; zero or many tags can reference one
-reference one manifest.</li>
+manifest.</li>
-<li>A <strong>manifest digest</strong> identifies exact manifest bytes. A multi-platform
+<li>A <strong>manifest digest</strong> identifies exact manifest bytes; a multi-platform
-<strong>index</strong> has its own digest; each platform manifest has another; layers and
+<strong>index</strong>, each platform manifest, and every layer and config blob carry their
-config blobs have their own again.</li>
+own.</li>
 <li><code>name:tag@digest</code> is client-side grammar, not wire protocol: the OCI
 distribution spec takes a tag <em>or</em> a digest in the URL. Clients resolve the digest
-and ignore the tag (Kubernetes documents exactly this), so digest-pinned pulls are immune
+and ignore the tag (Kubernetes documents this), so digest-pinned pulls are immune to tag
-to tag moves - a fact this page once mislearned as a cache trap and later retested: the
+moves. This page once mislearned the combined form as a cache-miss bug; retesting on
-historical zot rejection of combined references did not reproduce on v2.1.17. Traps are
+v2.1.17 showed the historical zot rejection no longer reproduces.</li>
 perishable; retest your scars.</li>
 <li>Digest pinning provides <strong>content integrity</strong> - not provenance, not
 authorisation, not availability, and not freedom from network fetches on a cold store.</li>
 </ul>
-<p>For a mirror, digests carry one sharp operational rule, learned here as a crashloop:
+<p>For a mirror, digests carry one sharp operational rule: zot converts Docker-schema
-zot converts Docker-schema manifests to OCI by default, and conversion changes the digest -
+manifests to OCI by default, and conversion changes the digest - breaking every digest pin
-silently breaking every digest pin and signature downstream. The pairing that prevents it
+and signature downstream. The pairing that prevents it (<code>preserveDigest: true</code>
-(<code>preserveDigest: true</code> per upstream, with <code>http.compat: ["docker2s2"]</code>)
+per upstream, with <code>http.compat: ["docker2s2"]</code>) is mandatory in both
-is mandatory in both directions: the registry refuses to start with one and not the other.
+directions: the registry refuses to start with one and not the other (the Loop 1 field
-This deployment runs both, on all five upstreams - observed in the live config.</p>
+note is this rule being learned). This deployment runs both on all five upstreams.</p>
-<p>Preserving digests does <em>not</em> mean the supply chain travels whole. OCI 1.1
+<p>Digest preservation and artifact completeness are different properties. The docs tie
-referrers - signatures, SBOMs, attestations attached via the <code>subject</code> field and
+<code>preserveDigest</code>/<code>compat</code> to keeping mirrored manifest bytes and
-served by the referrers API - are separate objects with their own discovery path. A
+media types - and therefore signature <em>validity</em> - aligned with upstream. Whether
-pull-through cache that mirrors manifests and blobs does not automatically carry them.
+signature, SBOM and attestation objects <em>arrive</em> is separate: OCI 1.1 referrers
-Nothing here verifies signatures today; if it did, disconnected verification would also
+ride the <code>subject</code> field and their own API, legacy Cosign signatures ride
-need the trust material carried locally (a public key, or a trusted-root bundle for a
+tag-schema conventions, and both depend on origin support and this version's sync
-private signing stack). That is stated as a boundary, not an aspiration.</p>
+behaviour. This deployment has not verified a complete referrer graph for any origin and
-
+performs no cryptographic signature verification; disconnected verification would also
-<div class="clause"><span class="k">THE HOMELAB CLAUSE</span><br>Production estates promote
+need locally held trust material. Both gaps sit in the open register. Production estates
-releases by digest between registries - a tag is a suggestion, a digest is a fact - and
+promote releases by digest and decide per policy whether referrers travel too; this lab
-decide explicitly whether referrers travel with them. The lab runs the digest half of that
+runs only the digest half.</p>
 discipline; the referrer half is future work and labelled as such.</div>
 <h2 id="retention"><span class="n" aria-hidden="true">06</span>Retention, GC and the digest-only trap</h2>
@@ -512,35 +491,29 @@ discipline; the referrer half is future work and labelled as such.</div>
  <div class="cap">Cached bricks queued behind a jammed intake: availability problems here are quiet, not loud.</div>
 </div></div>
-<p>This section corrects the largest error in this page's own first edition. I wrote:
+<p>This section corrects the largest error in this page's first edition, which said "no
-"no retention config means keep everything". That is true for tags, and dangerously false
+retention config means keep everything". True for tags; false for everything else - and
-for everything else - and "everything else" includes precisely the content this mirror
+"everything else" includes the content this mirror exists to hold. The semantics, from
-exists to hold.</p>
+upstream documentation and the deployed version's source:</p>
 <p>The verified semantics, from upstream documentation and the deployed version's source:</p>
 <ul>
-<li>With no retention configuration, <strong>all tags are retained</strong> - and
+<li>With no retention configuration, <strong>all tags are retained</strong> and
-<strong>all untagged manifests are deleted</strong> by garbage collection (unless referenced
+<strong>all untagged manifests are deleted</strong> by GC (unless referenced by an index
-by an index or artifact) once they are older than <code>gcDelay</code>. This deployment's
+or artifact). Timing is two-stage: the manifest ages past <code>gcDelay</code> (1h here),
-delay is the 1-hour default, with GC sweeping daily.</li>
+then deletion happens at the next sweep (<code>gcInterval</code>, 24h here) - the survival
 window depends on where creation falls relative to that sweep, not a fixed one-hour fuse.</li>
 <li><strong>Digest-only pull-through entries are stored as untagged manifests.</strong>
-That is the exact wording of the upstream fix's problem statement: pull an image through
+That is the upstream fix's own problem statement: pull an image through the mirror by
-the mirror by digest, and the cache entry has no tag to protect it.</li>
+digest, and the cache entry has no tag to protect it.</li>
 </ul>
-<div class="sayit"><span class="k">SAY IT</span>A digest-pinned fleet, pulling through a
+<p>Combined with section 04's revalidation behaviour: <em>tag</em> pulls are cached but
-mirror whose GC treats digest-only entries as garbage: the cache evicts exactly what the
+contact upstream anyway; <em>digest</em> pulls serve locally but their entries are
-estate is built on.</div>
+GC-eligible between sweeps - a digest-pinned fleet caches exactly the entry class GC may
 delete. On v2.1.17 this mirror is a rate-limit shield and a latency win, <strong>not yet a
 disconnection shelf</strong>; the first edition implied otherwise and was wrong.</p>
-<p>Put the pieces together with section 04's revalidation fact and the deployed behaviour
+<div class="diagram" role="region" tabindex="0" aria-label="Storage and retention diagram"><span class="k">Diagram C &middot; what keeps an object alive</span>
 is this: <em>tag</em> pulls are cached but always phone upstream anyway; <em>digest</em>
 pulls are served locally but their cache entries are eligible for eviction within hours.
 On v2.1.17, this mirror is a rate-limit shield and a latency win - it is <strong>not yet a
 disconnection shelf</strong>, and the first edition of this page was wrong to imply
 otherwise.</p>
 <div class="diagram"><span class="k">Diagram C &middot; what keeps an object alive</span>
 <svg viewBox="0 0 860 360" role="img" aria-labelledby="dc-t dc-d">
  <title id="dc-t">The OCI object graph and which references protect content from garbage collection</title>
  <desc id="dc-d">A tag points to an index; the index references platform manifests; manifests reference layers and a config blob. Referrers attach to a manifest via a subject field. A separate digest-only manifest sits with no tag pointing at it; it is eligible for garbage collection after the delay on the deployed version. Retention rules that keep tags do not protect the untagged manifest.</desc>
@@ -564,16 +537,16 @@ otherwise.</p>
    <rect x="30" y="215" width="280" height="52" rx="8" fill="none" stroke="#e8b44a"/><text x="170" y="236" fill="#dfe6f0">digest-only cached manifest</text><text x="170" y="253" fill="#e8b44a">NO tag references it</text>
    <rect x="380" y="215" width="220" height="52" rx="8" fill="none" stroke="#ff8fa3"/><text x="490" y="236" fill="#dfe6f0">GC after gcDelay (1h here)</text><text x="490" y="253" fill="#ff8fa3">deleteUntagged default: true</text>
    <line x1="310" y1="241" x2="375" y2="241" stroke="#ff8fa3" marker-end="url(#dc-a)"/>
-    <text x="430" y="310" fill="#98a2b4">keepTags rules protect TAGS in their repo. Nothing on v2.1.17/18 protects the amber box;</text>
+    <text x="430" y="310" fill="#98a2b4">keepTags protects TAGS in its repo. On v2.1.17/18 the amber box has one blanket lever: deleteUntagged:false,</text>
-    <text x="430" y="330" fill="#98a2b4">pull-aware keepUntagged ships in v2.1.19 (2026-08-04). Node-local caches can mask the eviction for days.</text>
+    <text x="430" y="330" fill="#98a2b4">which keeps ALL untagged manifests (capacity trade). Selective pull-aware keepUntagged ships in v2.1.19.</text>
  </g>
 </svg>
 <p class="legend">green = protected by a tag &middot; amber/red = the eviction path &middot; dashed magenta = referrers (separate lifecycle)</p>
-<p class="why">The lesson: reachability, not existence, is what GC respects - and a
+<p class="why">GC evaluates reference reachability: an untagged digest-only manifest can be
-pull-through cache full of digest pulls is a graveyard of unreachable-by-tag objects.</p></div>
+deleted even while its blobs remain present, and node-local image stores can mask the
 eviction for days.</p></div>
-<p>Retention policy semantics, stated precisely because two nearby systems use opposite
+<p>Retention policy semantics - two nearby systems use opposite matching rules:</p>
 rules:</p>
 <ul>
 <li>Retention policies match per repository, <strong>first match wins</strong> - order the
@@ -589,24 +562,21 @@ match. Two adjacent config blocks, two opposite precedence rules. Label which on
 reasoning about.</li>
 </ul>
-<p><strong>Mitigations, honestly ranked:</strong> upgrade to v2.1.19+ and configure
+<p><strong>Mitigations, ranked:</strong> upgrade to v2.1.19+ for pull-aware
-<code>keepUntagged</code> with pull-activity rules (the designed fix); until then, widen
+<code>keepUntagged</code> (the designed fix, schema to be validated against the actual
-<code>gcDelay</code>/<code>retention.delay</code> so eviction pressure drops (the upstream
+binary before rollout); until then, widen <code>gcDelay</code>/<code>retention.delay</code>
-maintainer's own interim suggestion), or set <code>deleteUntagged: false</code> and accept
+(the upstream maintainer's interim suggestion), or set <code>deleteUntagged: false</code>,
-that the cache only grows - a capacity trade, not a free lunch. Disabling GC entirely
+which protects every untagged manifest at the cost of unbounded cache growth. Disabling GC
-trades eviction for guaranteed storage exhaustion with no reclaim path; it is listed here
+entirely swaps eviction for storage exhaustion with no reclaim path.</p>
 to be argued against.</p>
 <div class="note warn"><span class="k">SAFE VALIDATION - PROPOSED, NOT YET RUN</span><br>
-The eviction claim above is documented upstream and consistent with this config; it has not
+The eviction claim is documented upstream and consistent with this config; it has not been
-been reproduced in this lab yet. The safe reproduction, for a <em>disposable</em> zot
+reproduced here. The reproduction, in a <em>disposable</em> instance only: record version
-instance only: record version + sanitised config; pull an image by digest through it;
+and sanitised config; pull by digest; confirm the stored manifest is untagged; shorten GC
-confirm on the registry side that the stored manifest is untagged; shorten GC timers (in
+timers; observe the manifest across a sweep; then block upstream and re-pull from a clean
-the disposable instance only); observe the manifest before and after the sweep; then block
+runtime store, recording serve vs re-fetch. Two upstream warnings: the retention
-upstream and repeat the pull from a clean runtime store, recording whether the mirror
+verification tool executes orphan-blob GC for real even in dry-run, and local storage
-serves or re-fetches. Two warnings from upstream documentation: the retention verification
+requires the registry stopped. Never point retention experiments at live storage.</div>
 tool executes orphan-blob GC for real even in dry-run, and on local storage it requires the
 registry stopped. Never point retention experiments at live storage.</div>
 <h2 id="auth"><span class="n" aria-hidden="true">07</span>The authentication migration</h2>
@@ -618,22 +588,19 @@ registry stopped. Never point retention experiments at live storage.</div>
  <div class="cap">The proving gate: a node counts as migrated when an authenticated pull succeeds where an anonymous one cannot.</div>
 </div></div>
-<p>Target state: the mirror refuses anonymous pulls. Current state: anonymous read on, with
+<p>Target state: the mirror refuses anonymous pulls. Current state: anonymous read on,
-per-node credentials staged in machine configs (inert until each node's reboot - the
+per-node credentials staged inert in machine configs until each reboot. The sequencing
-documented behaviour). The migration is a sequencing problem with one trap at its centre:</p>
+problem has one central hazard: <strong>while anonymous read is on, an ordinary pull says
 nothing about auth</strong> - a node whose credentials never applied is served as an
 anonymous reader and false-passes the gate. The canary must live in a repository that
 denies anonymous read, so success can only mean an authenticated pull.</p>
-<p><strong>While anonymous read is on, an ordinary pull proves nothing about auth.</strong>
+<p>The design is upstream-supported: authorisation resolves per-repository policies by
-A node whose credentials never applied sends no authorisation header, gets served as an
+<strong>longest match</strong> (<code>**</code> is the default for anything unmatched), and
-anonymous reader, and false-passes the gate. The canary must live in a repository that
+maintainer guidance confirms anonymous and authenticated access evaluate independently. So
-denies anonymous read, so a successful pull can only mean an authenticated pull.</p>
+a <code>canary/**</code> entry granting named identities read, with no
-
+<code>anonymousPolicy</code>, denies anonymous on that path while the glob keeps the rest
-<p>That design is now upstream-validated rather than assumed: zot authorisation resolves
+of the shelf open. The staged policy:</p>
 per-repository policies by <strong>longest match</strong> - the most specific path pattern
 wins, and <code>**</code> is explicitly the default policy for anything unmatched. A
 <code>canary/**</code> entry that grants named identities read and carries no
 <code>anonymousPolicy</code> therefore denies anonymous on that path while the glob keeps
 the rest of the shelf open. Maintainer guidance confirms anonymous and authenticated access
 are evaluated independently. The staged policy:</p>
 <pre>"accessControl": {
  "repositories": {
@@ -664,23 +631,20 @@ buildah, so the caveat is noted rather than felt.</div>
 <ol>
 <li>Stage credentials in the node's machine config (inert; documented as requiring reboot).</li>
 <li>Reboot the node at a planned window.</li>
-<li><strong>Prove identity:</strong> pull the protected canary through the node's own
+<li><strong>Identity check:</strong> pull the protected canary through the node's own
-runtime. Success = authenticated (anonymous cannot); failure = 401, back to step 1. The
+runtime. Success = authenticated (anonymous cannot); 401 = back to step 1. The expected
-expected split, to be confirmed against observed responses when the test runs: 401 for
+split - 401 for missing/wrong credentials, 403 for a valid identity lacking the action -
-missing/wrong credentials, 403 for a valid identity lacking the action - upstream docs
+sits in the open register, since the docs state the 403 case only for OIDC identities.</li>
-state the 403 case explicitly only for OIDC identities, so the basic-auth 403 boundary is
+<li><strong>Interception check, separately:</strong> the canary is a direct mirror
-listed in the open-verification register rather than asserted.</li>
+reference, so it says nothing about whether <code>docker.io/...</code> references route
-<li><strong>Prove interception separately:</strong> the canary pull is a direct reference
+through the mirror. That needs an original-upstream reference pulled on a node with a cold
-to the mirror, so it cannot prove that <code>docker.io/...</code> references are being
+runtime store, correlated with the mirror's request log - node-local content would satisfy
-routed through the mirror at all. That second proof needs an original-upstream reference
+the pull without any network and fake a pass.</li>
-pulled on the node <em>plus</em> the mirror's request log showing it arrive - and a cold
+<li>Anonymous read comes off the glob only when every node passes both checks. Rollback is
-runtime store, because node-local content will satisfy the pull without any network and
+the previous config commit, named before the flip.</li>
 fake a pass.</li>
 <li>Only when every node passes both proofs does <code>anonymousPolicy</code> come off the
 glob. Rollback is the previous config commit, and it is named before the flip, not after.</li>
 </ol>
-<div class="diagram"><span class="k">Diagram E &middot; the migration ladder</span>
+<div class="diagram" role="region" tabindex="0" aria-label="Authentication migration diagram"><span class="k">Diagram E &middot; the migration ladder</span>
 <svg viewBox="0 0 860 210" role="img" aria-labelledby="de-t de-d">
  <title id="de-t">The authentication migration ladder with its per-node proof gate and rollback point</title>
  <desc id="de-d">Five stages: anonymous baseline, credentials staged inert, node reboot, the two-part proof - authenticated canary pull plus logged mirror interception - and the final anonymous-off flip with a named rollback commit. A failing node loops from the proof back to staging.</desc>
@@ -701,24 +665,27 @@ glob. Rollback is the previous config commit, and it is named before the flip, n
  </g>
 </svg>
 <p class="legend">the flip is per-fleet; the proof is per-node - one unproven node under a fleet-wide flip is an outage wearing a green tick</p>
-<p class="why">Why two proofs: identity and interception fail independently, and each has a
+<p class="why">Identity and interception fail independently; each has a false-pass mode
-false-pass mode the other cannot detect.</p></div>
+the other cannot detect.</p></div>
-<p>Behind the gate sit five separate trust domains, deliberately not shared: node pull
+<p>Five trust domains never share material: node pull credentials (machine config,
-credentials (machine config, reboot-bound), CI push credentials (human-held login), the
+reboot-bound), the push credential (used by the operator-run build host for registry
-mirror's own upstream credentials (one mounted secret, pod-restart-bound), TLS trust
+logins, not embedded in CI configuration), the mirror's upstream credentials (one mounted
-(estate wildcard, standard roots), and human admin access (SSO in front of the UI). The
+secret, pod-restart-bound), TLS trust, and human admin access (SSO in front of the UI).
-push credential carries this page's oldest scar: its hash and plaintext once drifted apart,
+The push credential carries this page's oldest incident: in July 2026 its bcrypt hash and
-rotation became impossible, and pushes stayed dead for twelve days until the rebuilt flow
+plaintext drifted apart in the secrets manager, rotation became impossible, and pushes
-landed - three applications shipped the same day it came back. Hash and plaintext now live
+stayed dead for twelve days - three applications shipped the day the rebuilt flow landed.
-side by side, rotated together by a human, and the automation identity that syncs secrets
+Today the hash and plaintext are separate entries in the same secrets manager, rotated as
-is read-only by design so it can never half-rotate them again.</p>
+a pair by a human; the registry consumes only the derived htpasswd file, each plaintext
 reaches only its consumer, and the sync identity is read-only so automation cannot
 half-rotate the pair. Residual cost, stated: one secrets-manager project holds both
 halves, so its compromise yields verifier and credential together - versioned,
 consumer-scoped secret objects remain a possible refinement.</p>
 <h2 id="observability"><span class="n" aria-hidden="true">08</span>Observability that means something</h2>
-<p>"The registry is up" is the least useful sentence in this page. The worst incident in
+<p>The worst incident in this mirror's life: an in-flight sync wedged, every pull from
-this mirror's life: an in-flight sync wedged, every pull from that upstream hung, and the
+that upstream hung, and the health endpoint returned 200 throughout:</p>
 health endpoint returned 200 throughout. The log line was almost poetic:</p>
 <div class="pane-k">Replay &middot; observed 2026-07-07 &middot; the stalled sync</div>
 <pre class="pane">"image already demanded, waiting on channel"
@@ -729,30 +696,27 @@ $ curl -sfS --max-time 10 -o /dev/null -w '%{http_code}\n' \
    -H 'Accept: application/vnd.oci.image.index.v1+json'
 <span class="good">200</span>                                           <span class="cm">&lt;- THIS is "the mirror works"</span></pre>
-<p>Honesty about that incident, upgraded by reading the source: the log line is
+<p>The deployed source refines that story: the log line is coalescing by design -
-<em>coalescing by design</em> - concurrent requests for one image join the first sync
+concurrent requests join the first sync, which runs on a detached background context with
-rather than duplicating it, and since the deployed version syncs run on a detached
+a three-hour default timeout and survives client disconnects. Waiters blocking on a
-background context with a three-hour default timeout, surviving client disconnects.
+stalled sync until that timeout matches what we saw; the restart was recovery, and the
-Waiters blocking on a genuinely stalled sync until that timeout is consistent with what we
+root cause was never isolated. Two knobs this config leaves unset - sync
-saw; the restart was a recovery action, and the root cause was never isolated. Two knobs
+<code>maxRetries</code> (disabled by default upstream) and a tighter
-this config does not currently set - sync <code>maxRetries</code> (disabled by default
+<code>syncTimeout</code> - are the first change on recurrence.</p>
 upstream) and a tighter <code>syncTimeout</code> - are the first things a recurrence should
 change.</p>
-<p>What this deployment actually exposes, verified against the running version's source:
+<p>From the running version's source: <code>/livez</code>, <code>/readyz</code> and
-<code>/livez</code>, <code>/readyz</code> and <code>/startupz</code> exist as real health
+<code>/startupz</code> exist as health endpoints (absent from this version's docs),
-endpoints (absent from the docs at this version - a documentation gap, not an invention),
+alongside the spec's <code>/v2/</code>. The Prometheus metrics extension exists upstream -
-alongside the spec's <code>/v2/</code>. The Prometheus metrics extension exists upstream
+series names include <code>zot_http_requests_total</code>,
-(real series names include <code>zot_http_requests_total</code>,
+<code>zot_repo_storage_bytes</code>, <code>zot_repo_downloads_total</code> and
-<code>zot_repo_storage_bytes</code>, <code>zot_repo_downloads_total</code>,
+<code>zot_storage_lock_latency_seconds</code> - and is <strong>not enabled here</strong>;
-<code>zot_storage_lock_latency_seconds</code>) - and is <strong>not enabled in this
+wiring it, with scrape auth, is on the open list.</p>
 deployment</strong>. Wiring it, with scrape auth, is on the open list; no dashboard here
 pretends otherwise.</p>
-<p>The layered checks that would actually mean something, each answering one question:</p>
+<p>The layered checks, one question each:</p>
-<div class="tablewrap"><table>
+<div class="tablewrap" role="region" tabindex="0" aria-label="Layered operational checks"><table>
-<tr><th>Check</th><th>Question it answers</th><th>Status here</th></tr>
+<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Layered checks - one question each</caption>
 <tr><th scope="col">Check</th><th scope="col">Question it answers</th><th scope="col">Status here</th></tr>
 <tr><td><code>/livez</code> / <code>/readyz</code></td><td>is the process up / initialised</td><td><span class="st ok">available</span></td></tr>
 <tr><td>authenticated manifest GET of a local-only canary</td><td>can the right identity read real content from local storage</td><td><span class="st prop">proposed</span></td></tr>
 <tr><td>anonymous GET of the canary expecting 401</td><td>is the protection actually protecting</td><td><span class="st prop">proposed</span></td></tr>
@@ -762,12 +726,12 @@ pretends otherwise.</p>
 <tr><td>certificate expiry, credential age</td><td>what breaks on a schedule</td><td><span class="st prop">proposed</span></td></tr>
 </table></div>
-<p>The test matrix for the caching claims - each cell is an experiment, not an assumption
+<p>The caching test matrix (digest rows: source-verified and consistent with behaviour
-(status: the two digest rows are documented upstream and consistent with observed
+here; disconnected rows: not yet run in this lab):</p>
 behaviour here; the disconnected rows have not been run in this lab):</p>
-<div class="tablewrap"><table>
+<div class="tablewrap" role="region" tabindex="0" aria-label="Cache behaviour test matrix"><table>
-<tr><th>Scenario</th><th>Expected on v2.1.17</th><th>What proves it</th></tr>
+<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Cache behaviour - each row is an experiment</caption>
 <tr><th scope="col">Scenario</th><th scope="col">Expected on v2.1.17</th><th scope="col">What proves it</th></tr>
 <tr><td>cached tag, upstream reachable</td><td>served; upstream contacted anyway (revalidation)</td><td>zot log shows upstream request; latency includes round-trip</td></tr>
 <tr><td>cached tag, upstream unreachable</td><td>fails or stalls until timeout, then local fallback path</td><td>pull timing vs sync timeout; error text</td></tr>
 <tr><td>cached digest, upstream reachable</td><td>served locally, no upstream contact</td><td>absence of upstream request in zot logs during the pull</td></tr>
@@ -784,38 +748,33 @@ behaviour here; the disconnected rows have not been run in this lab):</p>
  <div class="cap">A central curated source and site-local registries: each site holds its own shelf, so a cut WAN idles nothing.</div>
 </div></div>
-<p>The first edition of this section asserted what "production" does. That was the wrong
+<p>The first edition of this section asserted what "production" does; production chooses
-register: production chooses from patterns against requirements. The honest version:</p>
+from patterns against requirements:</p>
 <ul>
-<li><strong>Single registry, accepted failure domain</strong> - exactly this lab's shape.
+<li><strong>Single registry, accepted failure domain</strong> - this lab's shape.
-Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or to a paused
+Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or a paused
-deploy window (fallback off) and that cost is accepted in writing.</li>
+deploy window (fallback off), with the cost accepted in writing.</li>
-<li><strong>Sync-based HA</strong> - upstream documents active/standby and active/active
+<li><strong>Sync-based HA</strong> - documented active/standby or active/active pairs of
-pairs of <em>independent</em> zot instances, each with its own storage, mirroring each
+<em>independent</em> instances, each with its own storage, mirroring each other behind a
-other behind a load balancer; the documented caveat is the synchronisation window between
+load balancer. The caveat: the synchronisation window between polls bounds what failover
-polls, which bounds what a failover can lose.</li>
+can lose.</li>
-<li><strong>Scale-out clustering</strong> - upstream's other documented shape: instances
+<li><strong>Scale-out clustering</strong> - instances shard repositories by hash and proxy
-shard repositories by hash and proxy to the owner, classically over shared S3-compatible
+to the owner, classically over shared S3-compatible storage. Horizontal scale for load;
-storage. That is horizontal scale for load; the shared storage is then the availability
+the shared storage becomes the availability story - a different property from the HA pair.</li>
-story, which is a different property than the HA pair above. Naming which property you are
+<li><strong>Edge estates</strong> - a central source promotes releases by digest; each
-buying is the design act.</li>
+site runs a local registry holding its release set on a schedule, so a severed WAN idles
-<li><strong>Edge estates</strong> - a central curated source promotes releases by digest;
+nothing at pull time. A disconnection claim is valid only when content, required
-each site runs a small local registry holding its release set, synced on a schedule, so a
+referrers, trust roots, credentials and the registry's own bootstrap image are already
-severed WAN idles nothing at pull time. A disconnection claim is only valid when the
+local (diagram D's right-hand column, per site).</li>
 required content, its referrers if policy demands them, trust roots, credentials and the
 registry's own bootstrap image are already local - the checklist from diagram D, applied
 per site.</li>
 </ul>
-<p>Whatever the topology: recovery time and recovery point get numbers before an incident
+<p>Whatever the topology: recovery time and point objectives get numbers before an
-provides them; backups are restore-tested copies off the failure domain (array snapshots
+incident supplies them; backups are restore-tested copies off the failure domain (array
-and RAID protect against disks, not against the array, the site, or an operator error -
+snapshots and RAID protect against disk loss, not against the array, the site or operator
-they are inputs to a backup strategy, not the strategy); capacity is planned against the
+error); cache capacity is planned from the section 06 retention policy; and the section 04
-retention policy from section 06, because "how big does the cache get" is a policy output,
+circular dependencies get drawn per site, because each exists at every scale - only the
-not a guess; and the circular dependencies from section 04 are drawn per site, because
+cost of ignoring them changes.</p>
 every one of them exists at every scale - the only thing that changes is how expensive
 they are to ignore.</p>
 <h2 id="verified"><span class="n" aria-hidden="true">10</span>Verified, open, and where the claims come from</h2>
@@ -824,13 +783,14 @@ they are to ignore.</p>
  <div class="cap">Proof of life is one authenticated byte transfer through the real path - not a status page.</div>
 </div></div>
-<p>When a node powers on, this fleet's gate makes it pull one known, first-party image
+<p>When a node powers on, the fleet's admission check performs an authenticated manifest
-through its own runtime before it counts as a member. That gate is the whole page in
+pull through the node's own runtime before the node counts as a member: a process-health
-miniature: prove the path, not the process.</p>
+response validates neither routing nor authorisation. Most of this page condenses into
 that one command.</p>
-<div class="tablewrap"><table>
+<div class="tablewrap" role="region" tabindex="0" aria-label="Architecture decision table"><table>
 <caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Architecture decisions</caption>
-<tr><th>Decision</th><th>Chosen</th><th>Why</th><th>Trade-off</th><th>Rollback / alternative</th><th>Status</th></tr>
+<tr><th scope="col">Decision</th><th scope="col">Chosen</th><th scope="col">Why</th><th scope="col">Trade-off</th><th scope="col">Rollback / alternative</th><th scope="col">Status</th></tr>
 <tr><td>Origin fallback</td><td>ON (default kept)</td><td>bootstrap survivability, Loop 1</td><td>silent mirror bypass possible</td><td>skipFallback:true + preseed (airgap shape)</td><td><span class="st ok">observed</span></td></tr>
 <tr><td>Sync mode</td><td>onDemand, all upstreams</td><td>cache follows real usage; Hub-safe</td><td>tag pulls revalidate upstream; digest entries untagged</td><td>polled sync for a curated release set</td><td><span class="st ok">observed</span></td></tr>
 <tr><td>Digest preservation</td><td>preserveDigest + docker2s2, x5</td><td>digest pins + signatures survive mirroring</td><td>mandatory config pairing (crashloop scar)</td><td>none - non-negotiable for digest-pinned estates</td><td><span class="st ok">observed</span></td></tr>