exploded-cluster/pilot/mirror.tpl.html

<style>
  /* Committed single-theme: a dark cinema world. Ground painted explicitly. */
  :root{
    --void:#070b14; --void-2:#0a0f1b; --ink:#dfe6f0; --ink-dim:#8b95a7;
    --cyan:#3fbaf5; --magenta:#e879f9; --line:#1c2434; --card:#0d1424;
    --mono:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;
    --sans:ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
  }
  *{box-sizing:border-box}
  html{scroll-behavior:smooth}
  body{margin:0;background:var(--void);color:var(--ink);
    font-family:var(--sans);font-size:1.06rem;line-height:1.65}
  #bar{position:fixed;top:0;left:0;height:2px;width:0;background:linear-gradient(90deg,var(--cyan),var(--magenta));z-index:9}
  .wrap{max-width:46rem;margin:0 auto;padding:0 1.25rem}
  header.hero{min-height:92vh;display:flex;flex-direction:column;justify-content:center;padding:4rem 0 2rem}
  .eyebrow{font-family:var(--mono);font-size:.78rem;letter-spacing:.22em;color:var(--cyan);text-transform:uppercase}
  h1{font-family:var(--sans);font-stretch:semi-condensed;font-weight:700;font-size:clamp(3rem,9vw,5.2rem);
    line-height:.95;margin:.5rem 0 1rem;text-wrap:balance;letter-spacing:.01em}
  .outcome{max-width:34rem;color:var(--ink-dim);font-size:1.12rem}
  .outcome b{color:var(--ink);font-weight:600}
  .cue{margin-top:3rem;font-family:var(--mono);font-size:.75rem;color:var(--ink-dim);letter-spacing:.15em}
  .cue::after{content:"";display:block;width:1px;height:56px;margin-top:.6rem;
    background:linear-gradient(var(--cyan),transparent)}
  h2{font-family:var(--sans);font-weight:600;font-size:2rem;letter-spacing:.02em;
    margin:4.5rem 0 1rem;text-wrap:balance}
  h2 .n{color:var(--cyan);font-family:var(--mono);font-size:1.05rem;vertical-align:.35em;margin-right:.6rem;letter-spacing:.1em}
  h3{font-family:var(--sans);font-weight:600;font-size:1.3rem;margin:2.2rem 0 .6rem}
  p{margin:.9rem 0}
  .scene{margin:3rem calc(50% - 50vw);width:100vw;overflow:hidden;position:relative}
  .scene .inner{max-width:72rem;margin:0 auto;padding:0 1rem;position:relative}
  .scene img{width:100%;height:auto;display:block;border-radius:.4rem}
  .scene .cap{font-family:var(--mono);font-size:.72rem;color:var(--ink-dim);
    letter-spacing:.14em;text-transform:uppercase;text-align:center;margin-top:.7rem}
  .sayit{border-left:3px solid var(--magenta);margin:2rem 0;padding:.4rem 0 .4rem 1.1rem;
    font-size:1.28rem;font-family:var(--sans);font-weight:600;line-height:1.35;color:#fff}
  .sayit .k{display:block;font-family:var(--mono);font-size:.7rem;letter-spacing:.2em;color:var(--magenta);margin-bottom:.3rem}
  .note{background:var(--card);border:1px solid var(--line);border-left:3px solid var(--cyan);
    border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
  .note .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan)}
  .clause{background:linear-gradient(135deg,rgba(232,121,249,.07),transparent 60%);
    border:1px solid var(--line);border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
  .clause .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--magenta)}
  pre{background:#050810;border:1px solid var(--line);border-radius:.5rem;padding:1rem 1.1rem;
    overflow-x:auto;font-family:var(--mono);font-size:.83rem;line-height:1.55;color:#c8d2e0}
  pre.pane{border-left:3px solid var(--cyan)}
  pre .cm{color:#5a6478}
  pre .good{color:#7fe0a7}
  pre .bad{color:#ff8fa3}
  .pane-k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--ink-dim);
    text-transform:uppercase;margin:1.6rem 0 .4rem}
  .diagram{background:var(--card);border:1px solid var(--line);border-radius:.55rem;
    padding:1.2rem;margin:2rem 0;overflow-x:auto}
  .diagram .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan);display:block;margin-bottom:.8rem}
  .diagram svg{width:100%;height:auto;display:block}
  .diagram .why{font-size:.9rem;color:var(--ink-dim);margin:.8rem 0 0}
  ol{padding-left:1.3rem}
  ol li{margin:.7rem 0}
  ul{padding-left:1.2rem}
  ul li{margin:.55rem 0}
  em{color:#fff;font-style:italic}
  .traps p{margin:1.1rem 0}
  .traps b{color:#fff}
  footer{margin:5rem 0 4rem;border-top:1px solid var(--line);padding-top:1.4rem;
    color:var(--ink-dim);font-size:.88rem;font-family:var(--mono)}
  a{color:var(--cyan)}
  :focus-visible{outline:2px solid var(--cyan);outline-offset:2px}
  .s21-ring{position:absolute;inset:0;pointer-events:none;mix-blend-mode:screen;opacity:.5;
    background:conic-gradient(from 0deg at 50% 50%, transparent 0deg, rgba(63,186,245,.28) 24deg, transparent 60deg);}
  .s22-glow{position:absolute;width:24%;aspect-ratio:1;border-radius:50%;pointer-events:none;
    mix-blend-mode:screen;opacity:0;background:radial-gradient(circle,rgba(63,186,245,.5),transparent 65%)}
  .s23-pulse{position:absolute;left:34%;top:44%;width:26%;aspect-ratio:1;border-radius:50%;pointer-events:none;
    mix-blend-mode:screen;opacity:.25;background:radial-gradient(circle,rgba(232,121,249,.55),transparent 62%)}
  .cursor::after{content:"_";color:var(--cyan);animation:blink 1.1s steps(1) infinite}
  @keyframes blink{50%{opacity:0}}
  @media (prefers-reduced-motion: reduce){
    html{scroll-behavior:auto}
    .cursor::after{animation:none}
    .s21-ring,.s23-pulse{display:none}
    .s22-glow{opacity:.35}
  }
</style>

<div id="bar"></div>

<header class="hero wrap">
  <nav style="position:absolute;top:1.2rem;left:1.25rem;font-family:var(--mono);font-size:.75rem"><a href="/" style="text-decoration:none">&larr; index</a></nav>
  <div class="eyebrow">Dossier / 005 &middot; The Teaching Lab</div>
  <h1>The Mirror</h1>
  <p class="outcome"><b>Outcome</b> &mdash; a teardown of a pull-through container registry:
  why a homelab runs its own mirror, the four chicken-and-egg loops hiding inside it,
  and the scars that taught each rule. Everything on this page is running, or broke,
  in the lab that published it.</p>
  <div class="cue">SCROLL</div>
</header>

<div class="wrap">

<h2><span class="n">00</span>Cold open &mdash; the build that "hung"</h2>

<div class="scene" id="s20"><div class="inner">
  <img src="__M20__" alt="An armoured way-station machine exploded into seven floating parts: hexagonal hull, roof plate, cassette shelf of glowing bricks, three intake pods, an output nozzle." width="1600" height="873" decoding="async">
  <div class="cap">Scene 20 &middot; the way-station &middot; motion: dock-and-stow (assembles as you scroll)</div>
</div></div>

<p>Nobody sets out to run a registry. You inherit the need the day a CI build stalls
for no reason you can see.</p>
<p>The lab's website builds from an ordinary Dockerfile. One base image came from
Docker Hub, pulled anonymously, and one evening the build simply... sat there. No
error visible. No progress. Here is the accurate version: the registry does not go
silent when it rate-limits you &mdash; it answers loudly, with an explicit 429 and the
string <code style="font-family:var(--mono)">toomanyrequests</code>. The <em>silence</em>
was manufactured closer to home, by a retry layer swallowing that answer and trying
again, backoff stacked on backoff, until a routine build read like a hung machine.
The swallower, recovered from the build script's own history, was a double retry
stack: an outer shell wrapper allowing four attempts with a growing backoff, wrapped
around the build tool's own three retries &mdash; up to twelve tries before anything
surfaced. It was added, in fairness, to ride out transient 502s; nets that catch 502s
catch 429s too, and a limit is not transient.</p>

<div class="pane-k">Read-only pane &middot; receipt 00a &middot; the silence, as committed to the repo</div>
<pre class="pane cursor">retry() {
  local n=0 max=4
  until "$@"; do
    n=$((n+1)); [ "$n" -ge "$max" ] &amp;&amp; { echo "&gt;&gt; failed after ${max} attempts" &gt;&amp;2; return 1; }
    echo "&gt;&gt; attempt ${n} failed, retrying in $((n*8))s..." &gt;&amp;2; sleep $((n*8))
  done
}
retry buildah build --retry 3 --retry-delay 5s ...
<span class="cm"># four attempts, each containing three - the 429 answered every one of them</span></pre>

<p>The cause was boring and invisible: anonymous pulls from the world's most popular
registry are rate-limited, and the base image tag was not pinned, so every single
build asked Docker Hub the same question again. The fix for THAT bug is nearly one
line &mdash; pin the digest. Nearly, because pinning only ends the conversation when a
local cache already holds the answer; an ephemeral runner with a cold store still
asks Hub for the manifest behind that digest every time. The pin stops the
re-resolution, not the round-trip. Which surfaced the sharper question:</p>

<div class="sayit"><span class="k">SAY IT</span>Why does a fleet of machines ask the
public internet for the same bytes, hundreds of times, forever?</div>

<p>The answer a production platform gives is a mirror: one machine that fetches an
image once, caches it, and serves every later request from the shelf. The lab built
one. This dossier takes it apart.</p>

<div class="note"><span class="k">FIELD NOTE</span><br>The mirror did not fix the red X
in CI. That red turned out to be a separate upstream bug in the CI system's log
finalise step &mdash; cosmetic, tolerated, documented. Two problems, one symptom.
Diagnose them separately or you will "fix" the wrong one and declare victory.</div>

<h2><span class="n">01</span>Anatomy &mdash; one cache, five upstreams</h2>

<p>The mirror here is zot: a single OCI registry, one namespace on the platform, one
50-gigabyte cache volume, digest-pinned like everything else in the fleet. It fronts
five public registries &mdash; Docker Hub, GitHub's, Quay, the Kubernetes project
registry, and NVIDIA's &mdash; and it works in one of three modes per upstream:</p>
<ul>
<li><b>onDemand</b> &mdash; pure pull-through. A miss fetches from upstream, caches, serves.
The second pull is instant and never leaves the building.</li>
<li><b>polled sync</b> &mdash; cache plus refresh on an interval, bounded by tag filters so a
poll cannot drag a whole vendor catalogue into the shelf.</li>
<li><b>pre-seed</b> &mdash; sync-only, for images that must be present before anything asks.</li>
</ul>
<p>Docker Hub gets onDemand ONLY. It has no catalogue API worth polling and the rate
limits that started this story punish enthusiasm.</p>
<p>The storage settings are not defaults; each one bought something:</p>

<pre>"storage": {
  "commit": true,
  "dedupe": true,
  "gc": true,
  "gcDelay": "1h",
  "gcInterval": "24h"
}</pre>

<p><code style="font-family:var(--mono)">commit</code> forces writes to disk before
acknowledging &mdash; crash-safety for a box that might lose power.
<code style="font-family:var(--mono)">dedupe</code> hard-links identical layers, which
matters enormously when five upstreams ship the same base layers under different
names. Garbage collection is delayed so a slow client mid-pull never watches its
blobs vanish.</p>

<p>The consumers point at the mirror in their runtime config. On this platform that
is a machine-level mirror list &mdash; and the FALLBACK SEMANTICS hiding behind this
short stanza are the most load-bearing thing in the whole dossier. Hold that thought
for chapter 02.</p>

<div class="diagram"><span class="k">Diagram D1 &middot; the pull path</span>
<svg viewBox="0 0 760 240" role="img" aria-label="Pull path: node asks mirror; hit serves from cache; miss fetches upstream then caches; mirror down falls through to origin">
  <defs><marker id="a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
  <marker id="am" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker></defs>
  <rect x="10" y="95" width="120" height="50" rx="8" fill="none" stroke="#3fbaf5"/><text x="70" y="125" fill="#dfe6f0" font-size="15" text-anchor="middle" font-family="IBM Plex Mono">node</text>
  <rect x="300" y="95" width="150" height="50" rx="8" fill="none" stroke="#3fbaf5"/><text x="375" y="118" fill="#dfe6f0" font-size="15" text-anchor="middle" font-family="IBM Plex Mono">mirror</text><text x="375" y="136" fill="#8b95a7" font-size="11" text-anchor="middle" font-family="IBM Plex Mono">(the shelf)</text>
  <rect x="600" y="30" width="150" height="50" rx="8" fill="none" stroke="#8b95a7"/><text x="675" y="60" fill="#dfe6f0" font-size="15" text-anchor="middle" font-family="IBM Plex Mono">origin</text>
  <line x1="130" y1="120" x2="295" y2="120" stroke="#3fbaf5" marker-end="url(#a)"/><text x="210" y="110" fill="#8b95a7" font-size="12" text-anchor="middle" font-family="IBM Plex Mono">pull</text>
  <path d="M450 108 Q 520 70 595 58" fill="none" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#a)"/><text x="520" y="62" fill="#8b95a7" font-size="12" text-anchor="middle" font-family="IBM Plex Mono">miss: fetch + cache</text>
  <path d="M375 145 Q 375 190 320 190 L 150 190 Q 130 190 130 150" fill="none" stroke="#3fbaf5" marker-end="url(#a)"/><text x="255" y="210" fill="#8b95a7" font-size="12" text-anchor="middle" font-family="IBM Plex Mono">hit: served from cache</text>
  <path d="M120 100 Q 300 20 595 42" fill="none" stroke="#e879f9" stroke-dasharray="3 5" marker-end="url(#am)"/><text x="300" y="30" fill="#e879f9" font-size="12" text-anchor="middle" font-family="IBM Plex Mono">mirror down: implicit fallback (skipFallback: false)</text>
</svg>
<p class="why">The decision this explains: the fallback is the platform's implicit
default &mdash; the design choice is refusing to disable it.</p></div>

<pre>machine:
  registries:
    mirrors:
      docker.io:
        endpoints:
          - https://zot.bztmon.org
      ghcr.io:
        endpoints:
          - https://zot.bztmon.org</pre>

<p>Read that carefully: the mirror is the ONLY listed endpoint, and that is still not
a hard dependency &mdash; because this platform falls back to the origin registry
IMPLICITLY unless you set <code style="font-family:var(--mono)">skipFallback: true</code>.
The lab's first version of this config listed the upstream as an explicit second
endpoint; the same-day refinement removed it after realising it merely rendered a
duplicate of what the default already guaranteed. The lesson generalises: know which
of your safety nets are things you built, and which are defaults you merely have not
broken.</p>

<p>Upstream credentials &mdash; a Docker Hub login to lift rate limits, a vendor key for
the GPU registry &mdash; live in ONE mounted secret, delivered by the cluster's secret
operator under logical names, never in the runtime config. That split is not
tidiness; it follows a reload asymmetry worth stating precisely. On this platform,
the mirror ENDPOINT list hot-reloads &mdash; apply it and the runtime picks it up live,
no reboot. Node-level registry AUTH does not: credentials in the machine config take
effect only after a reboot. The mounted secret rotates with a restart of one pod. So
endpoints and pod-side secrets can move cheaply; node auth is the expensive one &mdash;
and that asymmetry, not neatness, decides where each credential lives and the whole
rollout order in chapter 02. Both behaviours were observed on Talos v1.13.4 with
containerd 2.2.4 (June&ndash;July 2026); treat the exact split as version-dependent and
re-test on upgrades.</p>

<div class="clause"><span class="k">THE HOMELAB CLAUSE</span><br>In production this is
not one pod with a volume. The mirror sits on dedicated storage &mdash; a NAS-class
array or object store &mdash; sized for the whole estate's catalogue, and it is itself
backed up, because in a disconnected site the mirror IS the software supply. The
lab's 50Gi local volume with a nightly backup is the same organ, one size smaller.</div>

<h2><span class="n">02</span>Chicken and egg &mdash; four loops, four answers</h2>

<div class="scene" id="s21"><div class="inner">
  <img src="__M21__" alt="Six machine parts arranged on an invisible ring around empty centre - a large cyan way-station at twelve o'clock and a quarter-scale magenta replica of it among the orbiting parts." width="1600" height="893" decoding="async">
  <div class="s21-ring" id="ring"></div>
  <div class="cap">Scene 21 &middot; the loop &middot; motion: ring-walk (the cycle turns as you scroll)</div>
</div></div>

<p>Every infrastructure service eventually meets the question: what do you depend on,
and what happens when you ARE the dependency? The mirror has four of these loops, and
each one got a different answer. This chapter is the reason this page exists.</p>

<h3>Loop 1 &mdash; the mirror's own image comes through the mirror</h3>
<p>zot runs as a container. That container's image lives on one of the very
registries zot mirrors. So when the node that hosts zot boots, it asks... zot. Which
is not running, because the node is booting.</p>
<p>The answer is the fallback you met in chapter 01: <b>mirror listed, origin
implicit</b>. The runtime tries the mirror, fails fast, and falls through to the real
registry because nobody set <code style="font-family:var(--mono)">skipFallback: true</code>.
The loop breaks itself, and the thing that breaks it is a platform default doing
quiet duty as a bootstrap protocol &mdash; the design decision here is the restraint of
not turning it off.</p>

<div class="sayit"><span class="k">SAY IT</span>The fallback is not a compromise. The
fallback IS the design.</div>

<div class="note"><span class="k">FIELD NOTE</span><br>This was proven by accident. A
config change shipped with a flag the registry refused to start with
(<code style="font-family:var(--mono)">preserveDigest</code> demands
<code style="font-family:var(--mono)">http.compat</code> &mdash; a real pairing rule,
learn it from this page instead of from the crash). The mirror crashlooped. And
nothing else broke: every node quietly fell through to upstream and the fleet never
noticed. An unplanned failover test, passed. The postmortem produced the standing
rule: run <code style="font-family:var(--mono)">zot verify</code> on the config BEFORE
merging, in a throwaway pod, every time.</div>

<h3>Loop 2 &mdash; the recovery tooling deliberately ignores the mirror</h3>
<p>The fleet's rescue tooling &mdash; the automation that health-checks and rebuilds the
platform &mdash; could pull its execution image through the mirror like everything else.
It does not. Its image is cached on the operations box, outside the cluster entirely.</p>
<p>Because a recovery tool that depends on the thing it recovers is not a recovery
tool. It is a passenger.</p>

<h3>Loop 3 &mdash; you cannot turn authentication on everywhere at once</h3>

<div class="scene" id="s22"><div class="inner" style="position:relative">
  <img src="__M22__" alt="Four parts left to right: a waiting keyed cartridge, a small glowing key wedge, a tall gate frame with cyan inner glow, and a cartridge beyond the gate with its keyway lit." width="1600" height="873" decoding="async">
  <div class="s22-glow" style="left:16%;top:30%" data-band="0"></div>
  <div class="s22-glow" style="left:44%;top:28%" data-band="1"></div>
  <div class="s22-glow" style="left:70%;top:26%" data-band="2"></div>
  <div class="cap">Scene 22 &middot; the proving gate &middot; motion: checkpoint procession (three beats, left to right)</div>
</div></div>

<p>The target state is a mirror that refuses anonymous pulls. The path there is a
sequencing puzzle:</p>
<ol>
<li>Auth in the node's registry config requires a REBOOT to take effect &mdash; mirror
endpoints reload live, credentials do not. (Asymmetries like this decide rollout
order; find them before you start.)</li>
<li>The runtime's fall-back-on-401 behaviour is unreliable enough to carry open
upstream issues &mdash; so a node whose auth is wrong does not gracefully degrade, it
just fails to pull.</li>
<li>Therefore: anonymous read STAYS ON while every node gets its auth config and its
reboot, and each node must PROVE itself before it counts. And here the gate hides a
trap of its own: while anonymous read is still on, pulling an ordinary repo proves
NOTHING about auth &mdash; a node whose credentials never applied sends no
authorisation header at all, the mirror happily serves it as an anonymous reader,
and the gate false-passes the exact node it exists to catch. The canary must
therefore live in a repo whose access policy DENIES anonymous read, so a successful
pull can only mean an authenticated pull. Pass THAT, and the node is trusted. Only
when every node has passed does anonymous flip off fleet-wide.</li>
</ol>

<p>The lab's live policy today is the pre-flip shape &mdash; one glob, anonymous read on
&mdash; and the canary repo gets its own entry carrying NO anonymous policy, so its
pulls demand credentials while the rest of the shelf stays open:</p>

<pre>"accessControl": {
  "repositories": {
    "**": {
      "anonymousPolicy": ["read"],
      "policies": [
        { "users": ["zot-push"], "actions": ["read", "create", "update"] },
        { "users": ["zot-pull"], "actions": ["read"] }
      ]
    },
    "canary/**": {
      "policies": [
        { "users": ["zot-pull", "zot-push"], "actions": ["read"] }
      ]
    }
  }
}</pre>

<p>That per-node proof is a hard gate precisely because the failure mode is silent. A
node that looks fine and cannot pull is a time bomb with a pleasant dashboard.</p>

<div class="diagram"><span class="k">Diagram D2 &middot; the auth-flip ladder</span>
<svg viewBox="0 0 760 200" role="img" aria-label="Five rungs: anonymous on, auth staged, reboot, authenticated canary proof, anonymous off. A failing node loops from the proof rung back to reboot.">
  <defs><marker id="b" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
  <marker id="bm" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker></defs>
  <g font-family="IBM Plex Mono" font-size="11" text-anchor="middle">
  <rect x="8"   y="70" width="128" height="46" rx="8" fill="none" stroke="#8b95a7"/><text x="72"  y="90" fill="#dfe6f0">anon read ON</text><text x="72" y="105" fill="#8b95a7">baseline</text>
  <rect x="163" y="70" width="128" height="46" rx="8" fill="none" stroke="#3fbaf5"/><text x="227" y="90" fill="#dfe6f0">auth staged</text><text x="227" y="105" fill="#8b95a7">inert in config</text>
  <rect x="318" y="70" width="128" height="46" rx="8" fill="none" stroke="#3fbaf5"/><text x="382" y="90" fill="#dfe6f0">node reboots</text><text x="382" y="105" fill="#8b95a7">auth goes live</text>
  <rect x="473" y="70" width="128" height="46" rx="8" fill="none" stroke="#e879f9"/><text x="537" y="90" fill="#dfe6f0">canary pull</text><text x="537" y="105" fill="#e879f9">MUST authenticate</text>
  <rect x="628" y="70" width="124" height="46" rx="8" fill="none" stroke="#3fbaf5"/><text x="690" y="90" fill="#dfe6f0">all proven:</text><text x="690" y="105" fill="#8b95a7">anon OFF</text>
  <line x1="136" y1="93" x2="158" y2="93" stroke="#3fbaf5" marker-end="url(#b)"/>
  <line x1="291" y1="93" x2="313" y2="93" stroke="#3fbaf5" marker-end="url(#b)"/>
  <line x1="446" y1="93" x2="468" y2="93" stroke="#3fbaf5" marker-end="url(#b)"/>
  <line x1="601" y1="93" x2="623" y2="93" stroke="#3fbaf5" marker-end="url(#b)"/>
  <path d="M537 116 Q 537 165 460 165 L 420 165 Q 382 165 382 121" fill="none" stroke="#e879f9" stroke-dasharray="4 4" marker-end="url(#bm)"/>
  <text x="462" y="185" fill="#e879f9">401: back to the reboot rung</text></g>
</svg>
<p class="why">The decision this explains: the gate is per-node, never fleet-wide &mdash;
one unproven node under a fleet-wide flip is an outage wearing a green tick.</p></div>

<div class="pane-k">Read-only pane &middot; replay 23a &middot; the crictl gate, two different nodes</div>
<pre class="pane">node A (auth applied, rebooted):
  $ crictl pull zot.bztmon.org/canary/prompt-forge:0726R1
  <span class="good">Image is up to date for sha256:...</span>      <span class="cm">&lt;- authenticated pull; node A is trusted</span>
node B (auth staged in config, no reboot yet):
  $ crictl pull zot.bztmon.org/canary/prompt-forge:0726R1
  <span class="bad">E... failed to pull ...: 401 Unauthorized</span>
                                          <span class="cm">&lt;- node B reboots before it counts</span></pre>

<h3>Loop 4 &mdash; the mirror's host is also the mirror's customer</h3>
<p>The node that hosts the mirror boots its own workloads through it &mdash; including,
in this lab, the tunnel that serves the public website. So "restart the mirror's
node" carries a blast radius far beyond the mirror. The runbook for that reboot
lists every public-facing thing that rides on it, and the order they come back.</p>
<p>Draw the dependency arrows for your own estate. The ones that surprise you are
the ones that will page you. Sequencing, here, is a first-class engineering
artefact.</p>

<div class="clause"><span class="k">THE HOMELAB CLAUSE</span><br>Production separates
these concerns physically: the mirror on its own storage appliance, per-site edge
nodes pulling FROM it, no workload sharing its host. The lab collapses them onto one
node and manages the consequence with sequencing and runbooks. Same physics,
different budget.</div>

<h2><span class="n">03</span>The traps &mdash; each rule has a scar</h2>

<div class="scene" id="s23"><div class="inner" style="position:relative">
  <img src="__M23__" alt="A serene queue: a stuck intake pod with a magenta ring, three cyan cargo bricks waiting in a diagonal line, and a small healthy pod far away." width="1600" height="873" decoding="async">
  <div class="s23-pulse" id="pulse"></div>
  <div class="cap">Scene 23 &middot; the jammed intake &middot; motion: the only breathing still on the page</div>
</div></div>

<div class="traps">
<p><b>"The registry is up" is not "the mirror works."</b> The worst incident in this
mirror's life: an in-flight upstream sync wedged, and from then on EVERY pull of ANY
image from that upstream hung &mdash; while the registry's health endpoint returned 200
the entire time. The log line, when found, was almost poetic:</p>
</div>

<div class="pane-k">Read-only pane &middot; replay 23b &middot; the deadlock</div>
<pre class="pane">"image already demanded, waiting on channel"
$ kubectl -n zot rollout restart deploy zot   <span class="cm">&lt;- the fix, instantly effective</span>
<span class="cm"># probe an actual manifest, not the health endpoint:</span>
$ curl -sI https://zot.bztmon.org/v2/library/busybox/manifests/latest
<span class="good">HTTP/2 200</span>                                    <span class="cm">&lt;- THIS is "the mirror works"</span></pre>

<div class="traps">
<p>Monitor the thing you actually need &mdash; a manifest fetch &mdash; not the process's
opinion of itself.</p>
<p><b>Digests can change in transit.</b> A mirror that converts image formats on sync
breaks every digest pin and signature downstream of it &mdash; silently. The pairing
that prevents it (<code style="font-family:var(--mono)">preserveDigest</code> +
<code style="font-family:var(--mono)">http.compat</code>) is exactly the pairing from
the chapter-02 crashloop. One decision, two scars.</p>
<p><b>The cache grows until the disk ends.</b> onDemand means every image anyone ever
pulls joins the shelf, and this registry does not evict least-recently-used blobs on
its own &mdash; garbage collection reclaims orphaned data, not old-but-valid images.
The receipt from the lab's live config: the storage section carries gc settings and
NO retention section at all. That is a real answer &mdash; the current policy is "keep
everything", chosen implicitly, and the 50-gigabyte volume has never yet been pushed
to its high-water mark, so what happens there remains genuinely unobserved. A volume
size is a budget, not a policy, and the two retention behaviours on this page &mdash;
this one and the keep-rule inversion below &mdash; are ONE open policy decision: say
explicitly what is kept, what is deletable, and what happens when the disk fills,
before the disk answers for you.</p>
<p><b>Retention flips its meaning on the first rule.</b> No retention policy: keep
everything. ONE keep-rule on one repo: everything not matching is now DELETABLE.
Adding a policy is not narrowing &mdash; it is inverting. Write explicit keep coverage
per repo pattern, or write none &mdash; and read this trap together with the
cache-growth one above; they are the two halves of the same unanswered question.</p>
<p><b>Dedupe is not a toggle.</b> Flipping it on existing storage triggers a full
relink pass under lock &mdash; pushes can hang half an hour behind it. Maintenance
window, or leave it alone.</p>
<p><b>A tag and a digest in the same reference used to miss the cache &mdash; and this
trap carries its own moral about traps.</b> The historical mechanism: the runtime
routes to mirrors by HOST, so the combined reference reached the mirror, which
rejected it as unresolvable ("repository name not known", upstream issue zot 2584),
and the runtime fell through to the origin &mdash; the fallback that saves chapter 02
quietly defeating the cache. Re-tested against this lab's current registry on
2026-08-24: the combined reference now pulls clean; the failure did not reproduce on
v2.1.17. The rule survives in weaker form &mdash; prefer one reference form, know your
version &mdash; but the stronger lesson is that a trap list is perishable. Re-run your
own scars occasionally; some of them have healed.</p>
<p><b>Credentials drift when the hash and the plaintext live apart.</b> The push
credential is stored hashed; the drift surfaced on the first of July &mdash; plaintext
lost, hash still standing &mdash; and with no way to prove or re-mint the pair, pushes
stayed dead until the rebuilt rotation flow landed on the thirteenth. Twelve days.
The receipt that it worked: three separate applications shipped to the fleet the
SAME day the credential came back. Now hash and plaintext live side by side in the
secret store, rotated together, by a human &mdash; the sync identity is deliberately
read-only and CANNOT write secrets, so no automation can half-rotate them again.</p>
<p><b>The cache is invisible in the UI.</b> The web interface shows pushed
repositories only; on-demand cached images do not appear. Cosmetic &mdash; but the
first time you look, you will conclude the mirror is empty. Verify with the API
(<code style="font-family:var(--mono)">/v2/&lt;name&gt;/tags/list</code>), not the browser.</p>
</div>

<h2><span class="n">04</span>The prod translation &mdash; same organ, grown up</h2>

<div class="scene" id="s24"><div class="inner">
  <img src="__M24__" alt="A monolithic armoured vault with a glowing wall of bricks visible through its parted door, attended by four small self-sufficient way-stations each holding its own bricks." width="1600" height="873" decoding="async">
  <div class="cap">Scene 24 &middot; the vault and the way-stations &middot; motion: two-plane parallax, calm authority</div>
</div></div>

<p>Everything above is one lab, one node, one cache. Scale the pattern honestly and
it becomes the supply chain of every disconnected or edge estate:</p>
<ul>
<li><b>The mirror lives on real storage</b> &mdash; an appliance with parity and
snapshots, because at a disconnected site the mirror is not an optimisation, it is
the only source of software that exists.</li>
<li><b>Each remote site runs its OWN small node</b>, pulling from the central mirror
on a schedule, so a site survives its WAN link dying with its full catalogue local.</li>
<li><b>Authentication is day one</b>, not phase two &mdash; the lab's staged flip is
what retrofitting looks like; greenfield does not get that luxury or that risk.</li>
<li><b>Promotion is by digest, not by tag.</b> Production estates rewrite image
references to pinned digests and promote by copying digests between registries &mdash;
a tag is a suggestion, a digest is a fact. The lab's digest-pinning discipline is
the same muscle at one-tenth the ceremony.</li>
</ul>

<div class="note"><span class="k">FIELD NOTE &middot; LOOP 1 DOES NOT SURVIVE THE AIRGAP</span><br>
The lab's prettiest trick &mdash; mirror listed, origin implicit, the loop that breaks
itself &mdash; assumes an upstream EXISTS to fall through to. At a genuinely
disconnected site there is no second endpoint; if the mirror's own image is not
already on the host, nothing will ever fetch it. Production answers this the
unglamorous way: the mirror's image is pre-seeded into the host's local image store
at provisioning time (or run as a static workload imported straight from disk), or
Loop 2's recovery-box pattern is applied to the mirror itself &mdash; its image kept
OFF the platform it serves. Loop 1's implicit fallback is a connected-world luxury;
the airgap makes you choose Loop 2's discipline whether you like it or not.</div>

<div class="diagram"><span class="k">Diagram D3 &middot; hub and spokes, WAN cut</span>
<svg viewBox="0 0 760 230" role="img" aria-label="A central vault serving three sites; one site's WAN link is severed and that site still serves locally from its own shelf.">
  <defs><marker id="c" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker></defs>
  <g font-family="IBM Plex Mono" font-size="11" text-anchor="middle">
  <rect x="320" y="20" width="120" height="60" rx="8" fill="none" stroke="#e879f9"/><text x="380" y="46" fill="#dfe6f0">central</text><text x="380" y="62" fill="#dfe6f0">vault</text>
  <rect x="60"  y="150" width="130" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="125" y="172" fill="#dfe6f0">site A node</text><text x="125" y="188" fill="#8b95a7">own shelf</text>
  <rect x="315" y="150" width="130" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="380" y="172" fill="#dfe6f0">site B node</text><text x="380" y="188" fill="#8b95a7">own shelf</text>
  <rect x="570" y="150" width="130" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="635" y="172" fill="#dfe6f0">site C node</text><text x="635" y="188" fill="#8b95a7">own shelf</text>
  <line x1="340" y1="80" x2="150" y2="145" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#c)"/>
  <line x1="380" y1="80" x2="380" y2="145" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#c)"/>
  <line x1="420" y1="80" x2="600" y2="140" stroke="#e879f9" stroke-dasharray="2 6"/>
  <text x="540" y="100" fill="#e879f9" font-size="14">&#10007;</text><text x="560" y="118" fill="#e879f9">WAN cut</text>
  <text x="635" y="222" fill="#7fe0a7">still serving, local shelf</text>
  <text x="230" y="120" fill="#8b95a7">scheduled sync</text></g>
</svg>
<p class="why">The decision this explains: per-site nodes exist so a cut link idles
NOTHING &mdash; pulls never cross the WAN at pull time at all.</p></div>

<div class="clause"><span class="k">THE HOMELAB CLAUSE, INVERTED</span><br>The lab is
not a toy version of production. It is production with one of everything. The loops
in chapter 02 exist at every scale; the only thing that changes is how expensive
they are to ignore.</div>

<h2><span class="n">05</span>Close &mdash; the canary</h2>

<div class="scene" id="s25"><div class="inner">
  <img src="__M25__" alt="A wide flat node slab in the lower third of an otherwise empty void; a single luminous brick hovers midway in its descent toward the slab; a small pod watches from the edge." width="1600" height="873" decoding="async">
  <div class="cap">Scene 25 &middot; the canary &middot; motion: none. Deliberately.</div>
</div></div>

<p>The mirror's proof-of-life is not a dashboard. When a node powers on, the fleet's
gate makes it pull one known, first-party, mirror-only image through its own runtime
&mdash; the same path a real workload would take, auth and all. Pass, and the node is a
member of the fleet. Fail, and it is a machine that happens to be switched on.</p>

<div class="sayit"><span class="k">SAY IT</span>A node is not "up" when it pings. A
node is up when it can feed itself.</div>

<p>That is the whole dossier in one sentence. A platform is a chain of machines
feeding each other bytes, and the mirror is the pantry. Stock it deliberately, prove
it constantly, and never let the recovery crew depend on it for lunch.</p>

<footer>
  DOSSIER / 005 &middot; rev 3 &middot; source of truth: the running lab &middot; one open
  verification: negative-test the canary repo before Phase B leans on it. Motion is
  plate-level v1; per-part sliced animation arrives in a later revision.
  <a href="/">Back to the index</a>
</footer>

</div>

<script src="__MIRROR_JS__"></script>