jwright/exploded-cluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b27417ff023d8b6772b8b230c14f0f4cdfb2c07b
exploded-cluster/pilot/mirror.tpl.html
T
jwright b67ada9c65 review pass: restore page tokens lost in the masthead refactor, revert wrap width, artefact register 
The shared-masthead change stripped the mirror page's local :root including
four tokens the shell does not define (card, void-2, good, bad), leaving card
backgrounds and status colours undefined on the live page. The index wrap was
also narrowed to 64ch, squeezing the chapter cards; the masthead carries its
own width limit so wrap returns to 56rem. Two American spellings in mirror
prose corrected to artefact.
2026-08-25 20:29:22 +10:00

835 lines
67 KiB
HTML
Raw Blame History

<style>
:root{--void-2:#0a0f1b;--card:#0d1424;--good:#7fe0a7;--bad:#ff8fa3}
*{box-sizing:border-box}
html{scroll-behavior:smooth}
body{margin:0;background:var(--void);color:var(--ink);
font-family:var(--sans);font-size:1.05rem;line-height:1.65}
#bar{position:fixed;top:0;left:0;height:2px;width:0;background:linear-gradient(90deg,var(--cyan),var(--magenta));z-index:9}
/* full-bleed via named grid lines - no viewport units, so the vertical scrollbar
can never make the document wider than the visible viewport */
.layout{display:grid;
grid-template-columns:[full-start] minmax(1.25rem,1fr) [main-start] min(46rem,calc(100% - 2.5rem)) [main-end] minmax(1.25rem,1fr) [full-end]}
.layout>*{grid-column:main}
.layout>.scene{grid-column:full}
.eyebrow{font-family:var(--mono);font-size:.78rem;letter-spacing:.22em;color:var(--cyan);text-transform:uppercase}
h1{font-family:var(--sans);font-weight:750;font-size:clamp(2.8rem,8vw,4.6rem);
line-height:.98;margin:.5rem 0 1rem;text-wrap:balance;letter-spacing:-.01em}
.chapnav{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;margin:0 0 2.4rem}
.chapnav a{color:var(--ink-dim);text-decoration:none;border-bottom:1px solid transparent;
padding-bottom:2px;transition:color .2s ease,border-color .2s ease}
.chapnav a:hover,.chapnav a:focus-visible{color:var(--cyan);border-color:var(--cyan)}
h1 em{font-style:normal;background:linear-gradient(100deg,var(--cyan),var(--magenta));
-webkit-background-clip:text;background-clip:text;color:transparent;padding-bottom:.12em}
.standfirst{font-size:clamp(1.15rem,2.4vw,1.5rem);color:var(--ink);font-weight:600;
margin:.2rem 0 1rem;letter-spacing:-.01em;max-width:34rem}
h2{font-family:var(--sans);font-weight:700;font-size:1.85rem;letter-spacing:0;
margin:4.2rem 0 1rem;text-wrap:balance;scroll-margin-top:2.5rem}
h2 .n{color:var(--cyan);font-family:var(--mono);font-weight:500;font-size:1rem;vertical-align:.35em;margin-right:.65rem;letter-spacing:.1em}
h3{font-weight:650;font-size:1.28rem;margin:2.2rem 0 .6rem;scroll-margin-top:2.5rem}
p{margin:.9rem 0}
code,.mono{font-family:var(--mono);font-size:.92em}
.scene{margin:3rem 0;overflow:hidden;position:relative}
.scene .inner{max-width:72rem;margin:0 auto;padding:0 1rem;position:relative}
.scene img{width:100%;height:auto;display:block;border-radius:.4rem}
.scene .cap{font-family:var(--mono);font-size:.74rem;color:var(--ink-dim);
letter-spacing:.1em;text-align:center;margin-top:.7rem;padding:0 1rem}
[tabindex="0"]:focus-visible{outline:2px solid var(--cyan);outline-offset:3px}
.note{background:var(--card);border:1px solid var(--line);border-left:3px solid var(--cyan);
border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
.note .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan)}
.note.warn{border-left-color:var(--amber)}
.note.warn .k{color:var(--amber)}
.clause{background:linear-gradient(135deg,rgba(232,121,249,.07),transparent 60%);
border:1px solid var(--line);border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
.clause .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--magenta)}
pre{background:#050810;border:1px solid var(--line);border-radius:.5rem;padding:1rem 1.1rem;
overflow-x:auto;font-family:var(--mono);font-size:.83rem;line-height:1.55;color:#c8d2e0}
pre.pane{border-left:3px solid var(--cyan)}
pre .cm{color:#8a95ab}
pre .good{color:var(--good)}
pre .bad{color:var(--bad)}
.pane-k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--ink-dim);
text-transform:uppercase;margin:1.6rem 0 .4rem}
.diagram{background:var(--card);border:1px solid var(--line);border-radius:.55rem;
padding:1.2rem;margin:2rem 0;overflow-x:auto}
.diagram .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan);display:block;margin-bottom:.8rem}
.diagram svg{width:100%;height:auto;display:block;min-width:560px}
.diagram .why{font-size:.9rem;color:var(--ink-dim);margin:.8rem 0 0}
.diagram .legend{font-family:var(--mono);font-size:.72rem;color:var(--ink-dim);margin:.6rem 0 0}
table{border-collapse:collapse;width:100%;font-size:.88rem;margin:1.4rem 0}
.tablewrap{overflow-x:auto;margin:1.4rem 0}
.tablewrap table{margin:0;min-width:640px}
th,td{border:1px solid var(--line);padding:.5rem .65rem;text-align:left;vertical-align:top}
th{font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;text-transform:uppercase;color:var(--cyan);background:var(--void-2)}
td .st{font-family:var(--mono);font-size:.7rem;letter-spacing:.06em;padding:.05rem .4rem;border-radius:.25rem;border:1px solid var(--line);white-space:nowrap}
.st.ok{color:var(--good);border-color:rgba(127,224,167,.4)}
.st.open{color:var(--amber);border-color:rgba(232,180,74,.4)}
.st.prop{color:var(--magenta);border-color:rgba(232,121,249,.4)}
.facts{border:1px solid var(--line);border-radius:.55rem;background:var(--card);
padding:1.1rem 1.3rem;margin:2.2rem 0;font-size:.92rem}
.facts dl{display:grid;grid-template-columns:max-content 1fr;gap:.35rem 1.2rem;margin:0}
.facts dt{font-family:var(--mono);font-size:.72rem;letter-spacing:.12em;text-transform:uppercase;color:var(--ink-dim);padding-top:.15rem}
.facts dd{margin:0}
ol{padding-left:1.3rem} ol li{margin:.6rem 0}
ul{padding-left:1.2rem} ul li{margin:.5rem 0}
em{color:#fff}
.traps p{margin:1.1rem 0}
.traps b,strong{color:#fff}
footer{margin:5rem 0 4rem;border-top:1px solid var(--line);padding-top:1.4rem;
color:var(--ink-dim);font-size:.88rem;font-family:var(--mono)}
a{color:var(--cyan)}
:focus-visible{outline:2px solid var(--cyan);outline-offset:2px}
.s21-ring{position:absolute;inset:0;pointer-events:none;mix-blend-mode:screen;opacity:.5;
background:conic-gradient(from 0deg at 50% 50%, transparent 0deg, rgba(63,186,245,.28) 24deg, transparent 60deg)}
.s22-glow{position:absolute;width:24%;aspect-ratio:1;border-radius:50%;pointer-events:none;
mix-blend-mode:screen;opacity:0;background:radial-gradient(circle,rgba(63,186,245,.5),transparent 65%)}
.s23-pulse{position:absolute;left:34%;top:44%;width:26%;aspect-ratio:1;border-radius:50%;pointer-events:none;
mix-blend-mode:screen;opacity:.25;background:radial-gradient(circle,rgba(232,121,249,.55),transparent 62%)}
.refs li{margin:.45rem 0;font-size:.92rem}
.outro{max-width:64ch;margin:0 auto;padding:2vh 24px 14vh;text-align:center;border:0}
.outro p{font-family:var(--mono);font-size:.7rem;letter-spacing:.12em;color:var(--ink-dim);line-height:1.9;margin:0}
@media (prefers-reduced-motion: reduce){
html{scroll-behavior:auto}
.s21-ring,.s23-pulse{display:none}
.s22-glow{opacity:.35}
}
</style>
<div id="bar"></div>
<div class="layout">
<header class="masthead">
<nav class="chapnav" aria-label="Site"><a href="/">&larr; The Teaching Lab</a></nav>
<p class="eyebrow">Chapter 02 &middot; The Mirror &middot; Dossier 005</p>
<h1>The cache was the easy part.<br><em>Now watch what it quietly throws away.</em></h1>
<p class="lede">A pull-through registry, taken apart: what the deployed version actually
guarantees, the retention behaviour that evicts digest-pinned content, and how each lesson
translates to production. Every claim carries an evidence label - including the corrections
to my own first draft. Scroll, and the machinery comes apart.</p>
<dl class="mastfacts">
<div><dt>Registry</dt><dd>zot v2.1.17</dd></div>
<div><dt>Upstreams</dt><dd>5, on-demand</dd></div>
<div><dt>Platform</dt><dd>Talos / containerd</dd></div>
<div><dt>Open tests</dt><dd>6</dd></div>
</dl>
<p class="scrollcue" aria-hidden="true">scroll<span></span></p>
</header>
<div class="facts">
<dl>
<dt>Registry</dt><dd>zot v2.1.17 (digest-pinned image; latest upstream release at last check: v2.1.20)</dd>
<dt>Platform</dt><dd>Talos Linux v1.13.4, containerd 2.2.4, single-node Kubernetes</dd>
<dt>Upstreams</dt><dd>5 (Docker Hub, GHCR, Quay, registry.k8s.io, NGC), all on-demand pull-through</dd>
<dt>Storage</dt><dd>50Gi local PVC; dedupe on, GC on (1h delay / 24h interval); no retention config</dd>
<dt>Auth state</dt><dd>anonymous read enabled; per-node pull credentials staged, not yet enforced</dd>
<dt>Fallback</dt><dd>origin fallback ON (platform default; deliberately not disabled)</dd>
<dt>Reviewed</dt><dd>2026-08-25: live cluster configuration read; upstream docs and v2.1.17 source checked; live runtime tests listed separately in the open register</dd>
<dt>Open items</dt><dd>pull-aware digest-entry retention (needs v2.1.19+; blanket alternatives are capacity-heavy), auth canary negative test, upstream namespace prefixes, disconnected-serve drill</dd>
</dl>
<p style="margin:.9rem 0 0;font-size:.8rem;color:var(--ink-dim);font-family:var(--mono)">
Evidence labels used below: <span class="st ok">lab</span> reproduced here with a dated receipt &middot;
<span class="st ok">config</span> present in the inspected config &middot;
<span class="st ok">source</span> established from the deployed version's source &middot;
<span class="st ok">docs</span> stated in version-appropriate documentation &middot;
<span class="st open">reported</span> described in an unresolved upstream issue &middot;
<span class="st prop">proposed</span> not deployed &middot; <span class="st open">open</span> not yet run</p>
</div>
<nav class="toc" aria-label="Section index">
<p class="kt">Index</p>
<ol>
<li><a href="#incident"><b>Section 01</b><span>The build that "hung"</span></a></li>
<li><a href="#anatomy"><b>Section 02</b><span>What the mirror does</span></a></li>
<li><a href="#routing"><b>Section 03</b><span>Routing and five upstreams</span></a></li>
<li><a href="#bootstrap"><b>Section 04</b><span>Bootstrap, fallback, failure</span></a></li>
<li><a href="#digests"><b>Section 05</b><span>Digests and supply chain</span></a></li>
<li><a href="#retention"><b>Section 06</b><span>Retention and the digest trap</span></a></li>
<li><a href="#auth"><b>Section 07</b><span>The authentication migration</span></a></li>
<li><a href="#observability"><b>Section 08</b><span>Observability that means something</span></a></li>
<li><a href="#production"><b>Section 09</b><span>The production translation</span></a></li>
<li><a href="#verified"><b>Section 10</b><span>Verified, open, references</span></a></li>
</ol>
</nav>
<h2 id="incident"><span class="n" aria-hidden="true">01</span>The build that "hung"</h2>
<div class="scene" id="s20"><div class="inner">
<img src="__M20__" alt="An armoured way-station machine exploded into parts: hexagonal hull, roof plate, a shelf of glowing bricks, intake pods and an output nozzle - the mirror as a physical machine.">
<div class="cap">The mirror as a machine: upstream intakes, one cache shelf, one serving nozzle.</div>
</div></div>
<p>I added zot after a CI pull hit Docker Hub's anonymous rate limit and the retry wrapper
hid the 429. At the time of the incident, this site's build-stage pull was anonymous.</p>
<p>Docker Hub does not go silent when it limits you. It answers with HTTP 429 and the error
code <code>toomanyrequests</code>. Under the
<a href="https://docs.docker.com/docker-hub/usage/pulls/">current published limits</a>,
unauthenticated users get 100 pulls per six hours per IPv4 address (or IPv6 /64); a
single-platform image counts as one pull, a multi-architecture image counts once per
architecture pulled, version checks do not count, and a <code>HEAD</code> request can read
the rate-limit headers without consuming a pull. A separate abuse limiter covers all
request types with its own 429 form. The <em>silence</em> in this incident was
manufactured on my side, by retries swallowing the answer.</p>
<p>The swallower, recovered from the build script's git history, was a double retry stack:
an outer shell wrapper allowing four attempts, wrapped around
<code>buildah build --retry 3</code>. Buildah's flag counts <em>retries</em> - one initial
try plus up to three more, applying to registry push/pull operations. Four outer attempts,
each containing up to four inner attempts: <strong>up to sixteen requests</strong> for one
repeatedly failing registry operation (a build with several pulls can produce more), each
billed against the limit it was trying to outlast. Retry amplification turns a throttle
into an outage and hides the evidence while doing it.</p>
<div class="pane-k">Receipt &middot; the retry stack, as committed to the repo</div>
<pre class="pane">retry() {
local n=0 max=4
until "$@"; do
n=$((n+1)); [ "$n" -ge "$max" ] &amp;&amp; { echo "&gt;&gt; failed after ${max} attempts" &gt;&amp;2; return 1; }
echo "&gt;&gt; attempt ${n} failed, retrying in $((n*8))s..." &gt;&amp;2; sleep $((n*8))
done
}
retry buildah build --retry 3 --retry-delay 5s ...
<span class="cm"># --retry 3 = one attempt + three retries; the wrapper multiplies that by four.</span>
<span class="cm"># A 429 answered every one of them, and the backoffs read as a hang.</span></pre>
<p>Pinning the base image by digest stopped the tag re-resolution but not the round-trip:
an ephemeral runner with a cold store still fetches the manifest behind that digest every
build. The repeated origin pulls justified a shared cache. This page takes it apart,
including the parts that turned out not to work the way I first believed.</p>
<div class="note"><span class="k">FIELD NOTE</span><br>The mirror did not fix the red X in
CI. That red was a separate upstream bug in the CI system's log-finalise step - cosmetic,
tolerated, documented. Two problems, one symptom; diagnose them separately.</div>
<h2 id="anatomy"><span class="n" aria-hidden="true">02</span>What the mirror actually does</h2>
<p>The mirror is zot: one registry, one 50Gi cache volume, digest-pinned like everything
it serves. Five public registries front it in <strong>on-demand pull-through</strong> mode -
a miss fetches and caches, a hit serves from the shelf. zot also offers polled mirroring
and pre-seeding; Docker Hub is on-demand only, per its documentation.</p>
<p>The storage block, from the live config:</p>
<pre>"storage": {
"rootDirectory": "/var/lib/registry",
"commit": true,
"dedupe": true,
"gc": true,
"gcDelay": "1h",
"gcInterval": "24h"
}</pre>
<p><code>commit</code> asks zot to commit writes to disk immediately instead of relying on
buffered flushing; it narrows the buffered-write window, while end-to-end power-loss
durability still depends on the filesystem, volume and disk. <code>dedupe</code> uses hard
links on local filesystem storage (remote backends implement it differently), saving
capacity when five upstreams ship the same base layers under different names - at the cost
of a startup reconciliation pass that matters operationally when toggled on existing data.
And <code>gc</code> with its two timers looks innocuous here; section 06 is about why it is
the most consequential block on this page.</p>
<div class="diagram" role="region" tabindex="0" aria-label="System context diagram"><span class="k">Diagram A &middot; system context</span>
<svg viewBox="0 0 860 400" role="img" aria-labelledby="da-t da-d">
<title id="da-t">System context: consumers, the mirror, its storage, five upstreams, and the trust boundaries between them</title>
<desc id="da-d">CI and Kubernetes nodes pull from the zot mirror over the LAN. The mirror stores content on a 50Gi volume and syncs on demand from five upstream registries across the internet boundary. Anonymous read is currently allowed inbound; outbound upstream credentials live in one mounted secret. The origin-fallback path from nodes directly to upstreams is shown dashed.</desc>
<defs><marker id="da-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
<marker id="da-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker></defs>
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
<rect x="12" y="60" width="150" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="87" y="82" fill="#dfe6f0">CI (buildah)</text><text x="87" y="99" fill="#98a2b4">bastion host</text>
<rect x="12" y="150" width="150" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="87" y="172" fill="#dfe6f0">k8s nodes</text><text x="87" y="189" fill="#98a2b4">containerd + local store</text>
<rect x="300" y="95" width="180" height="72" rx="8" fill="none" stroke="#dfe6f0"/><text x="390" y="122" fill="#dfe6f0">zot v2.1.17</text><text x="390" y="139" fill="#98a2b4">anon read (today)</text><text x="390" y="156" fill="#98a2b4">sync: onDemand x5</text>
<rect x="322" y="205" width="136" height="46" rx="8" fill="none" stroke="#98a2b4"/><text x="390" y="224" fill="#dfe6f0">50Gi PVC</text><text x="390" y="241" fill="#98a2b4">dedupe + GC 1h/24h</text>
<line x1="390" y1="167" x2="390" y2="200" stroke="#98a2b4"/>
<rect x="660" y="14" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="39" fill="#dfe6f0">docker.io</text>
<rect x="660" y="82" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="107" fill="#dfe6f0">ghcr.io</text>
<rect x="660" y="150" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="175" fill="#dfe6f0">quay.io</text>
<rect x="660" y="218" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="243" fill="#dfe6f0">registry.k8s.io</text>
<rect x="660" y="286" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="311" fill="#dfe6f0">nvcr.io</text>
<line x1="162" y1="86" x2="295" y2="115" stroke="#3fbaf5" marker-end="url(#da-a)"/>
<line x1="162" y1="176" x2="295" y2="145" stroke="#3fbaf5" marker-end="url(#da-a)"/>
<line x1="480" y1="110" x2="655" y2="40" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
<line x1="480" y1="120" x2="655" y2="100" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
<line x1="480" y1="131" x2="655" y2="168" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
<line x1="480" y1="142" x2="655" y2="235" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
<line x1="480" y1="153" x2="655" y2="300" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
<path d="M162 200 Q 390 330 660 315" fill="none" stroke="#e879f9" stroke-dasharray="2 6" marker-end="url(#da-m)"/>
<text x="360" y="322" fill="#e879f9">origin fallback (dashed = only when the mirror cannot serve)</text>
<line x1="580" y1="8" x2="580" y2="345" stroke="#1c2434"/>
<text x="580" y="368" fill="#98a2b4">internet boundary - outbound creds in ONE mounted secret</text>
<text x="230" y="45" fill="#98a2b4">LAN - anonymous read today; per-node creds staged</text>
</g>
</svg>
<p class="legend">solid = normal pull path &middot; dashed cyan = on-demand sync on miss &middot; dashed magenta = origin fallback</p>
<p class="why">One cache, two consumer classes; the two credential domains never mix.</p></div>
<p>Consumers point at it via the platform's machine-level registry config:</p>
<pre>machine:
registries:
mirrors:
docker.io:
endpoints:
- https://zot.bztmon.org
ghcr.io:
endpoints:
- https://zot.bztmon.org</pre>
<p>The mirror is the only listed endpoint, yet not a hard dependency: Talos tries
endpoints in order "and by default the last implicit endpoint is the original upstream
registry", unless <code>skipFallback: true</code>. An early version listed the upstream
explicitly as endpoint two; it was removed as redundant - origin fallback here is a Talos
default, not an endpoint this project maintains.</p>
<p>One asymmetry decides section 07's rollout order. Talos's
<a href="https://docs.siderolabs.com/talos/v1.13/reference/configuration/v1alpha1/config/">v1alpha1
machine-config reference</a> states for registry auth: "changes to the registry auth will
not be picked up by the CRI containerd plugin without a reboot" - matching what this lab
saw on v1.13.4. Mirror <em>endpoint</em> changes applied live here (same versions; the
docs are silent on that half, so it stays a lab observation, not a cross-version
guarantee). Secrets the registry consumes from a mount rotate with a pod restart. Put each
credential on the side that can move.</p>
<p>One pod, one PVC, one node is an accepted failure domain here, recorded as such;
section 09 covers what changes when the requirements do.</p>
<h2 id="routing"><span class="n" aria-hidden="true">03</span>Request routing and five upstreams</h2>
<p>Five registries feed one endpoint: when a node asks the mirror for
<code>pause:3.10</code>, how does it know which origin that means? Three facts:</p>
<ol>
<li><strong>containerd names the origin</strong> - mirror requests carry it as a query
parameter: <code>/v2/pause/manifests/3.10?ns=registry.k8s.io</code> (documented).</li>
<li><strong>zot v2.1.17 ignores it</strong> - no <code>ns</code> handling exists in the
deployed request path; using it for upstream selection is open feature request zot 4187.
The URL path alone selects the local repository.</li>
<li><strong>This deployment has no per-upstream prefixes</strong> - the live sync config
gives all five upstreams <code>prefix: "**"</code> and no <code>destination</code>. On a
miss, zot tries upstreams in config order until one has the path; the docs' own
multi-registry example instead gives each a distinct <code>destination</code>.</li>
</ol>
<div class="tablewrap" role="region" tabindex="0" aria-label="Upstream mapping table"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Upstream map - observed configuration</caption>
<tr><th scope="col">Origin</th><th scope="col">Local namespace</th><th scope="col">Outbound auth</th><th scope="col">Sync mode</th><th scope="col">Digest preserved</th><th scope="col">On miss</th><th scope="col">Status</th></tr>
<tr><td>docker.io</td><td rowspan="5">flat (no prefix - collision-ambiguous by construction)</td><td>Hub login (rate-limit lift)</td><td>onDemand</td><td>yes</td><td rowspan="5">upstreams tried in config order; first that resolves the path wins</td><td><span class="st ok">observed</span></td></tr>
<tr><td>ghcr.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
<tr><td>quay.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
<tr><td>registry.k8s.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
<tr><td>nvcr.io</td><td>$oauthtoken + key</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
</table></div>
<p>The flat namespace has not bitten because the origins use largely disjoint paths -
Hub's official images live under <code>library/</code> (the implicit prefix behind bare
names like <code>alpine</code>), NGC under <code>nvidia/</code>. Largely disjoint is a
probability: an organisation existing on both GHCR and Quay would collide, and config order
would pick the winner. Accepted risk, recorded; not a design.</p>
<div class="note warn"><span class="k">PROPOSED - NOT YET DEPLOYED</span><br>The safer
shape, documented on both halves: a distinct <code>destination</code> prefix per upstream
in zot (<code>/docker</code>, <code>/ghcr</code>, ...), with each Talos mirror pointed at
the prefixed path via <code>overridePath: true</code> (suppressing the automatic
<code>/v2</code> append). The requested path then names the origin and collisions become
impossible. Migration cost: every cached repository changes local path, so the cache
re-warms.</div>
<div class="diagram" role="region" tabindex="0" aria-label="Request flow diagram"><span class="k">Diagram B &middot; request flow</span>
<svg viewBox="0 0 860 330" role="img" aria-labelledby="db-t db-d">
<title id="db-t">Request flow from a node through containerd to the mirror, with hit, miss, revalidation and fallback branches</title>
<desc id="db-d">A pull begins at the node's local content store. On local miss, containerd asks the mirror. A cached digest request is served from mirror storage without upstream contact. A tag request is revalidated against the upstream even when cached, on the deployed version. A storage miss triggers on-demand sync from the matching upstream. If the mirror cannot serve, containerd falls back to the origin registry.</desc>
<defs><marker id="db-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
<marker id="db-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker>
<marker id="db-g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#7fe0a7"/></marker></defs>
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
<rect x="10" y="30" width="170" height="46" rx="8" fill="none" stroke="#3fbaf5"/><text x="95" y="49" fill="#dfe6f0">node content store</text><text x="95" y="66" fill="#98a2b4">hit = no network at all</text>
<rect x="10" y="130" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="95" y="155" fill="#dfe6f0">containerd -&gt; mirror</text>
<rect x="300" y="40" width="200" height="44" rx="8" fill="none" stroke="#7fe0a7"/><text x="400" y="58" fill="#dfe6f0">digest request, cached</text><text x="400" y="75" fill="#7fe0a7">served locally (verified)</text>
<rect x="300" y="120" width="200" height="44" rx="8" fill="none" stroke="#e8b44a"/><text x="400" y="138" fill="#dfe6f0">tag request, cached</text><text x="400" y="155" fill="#e8b44a">still revalidates upstream</text>
<rect x="300" y="200" width="200" height="44" rx="8" fill="none" stroke="#3fbaf5"/><text x="400" y="218" fill="#dfe6f0">storage miss</text><text x="400" y="235" fill="#98a2b4">on-demand sync + cache</text>
<rect x="640" y="120" width="200" height="44" rx="8" fill="none" stroke="#98a2b4"/><text x="740" y="138" fill="#dfe6f0">origin registry</text><text x="740" y="155" fill="#98a2b4">rate limits live here</text>
<line x1="95" y1="76" x2="95" y2="125" stroke="#3fbaf5" marker-end="url(#db-a)"/><text x="150" y="105" fill="#98a2b4">local miss</text>
<line x1="180" y1="140" x2="295" y2="66" stroke="#7fe0a7" marker-end="url(#db-g)"/>
<line x1="180" y1="150" x2="295" y2="142" stroke="#e8b44a" marker-end="url(#db-a)"/>
<line x1="180" y1="160" x2="295" y2="218" stroke="#3fbaf5" marker-end="url(#db-a)"/>
<line x1="500" y1="142" x2="635" y2="142" stroke="#e8b44a" stroke-dasharray="5 4" marker-end="url(#db-a)"/>
<line x1="500" y1="222" x2="640" y2="158" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#db-a)"/>
<path d="M95 170 Q 95 300 400 300 L 660 300 Q 740 300 740 168" fill="none" stroke="#e879f9" stroke-dasharray="2 6" marker-end="url(#db-m)"/>
<text x="400" y="290" fill="#e879f9">containerd fallback: mirror endpoints exhausted -&gt; origin direct</text>
</g>
</svg>
<p class="legend">green = fully local &middot; amber = the surprising branch on v2.1.17 &middot; magenta = fallback</p>
<p class="why">The amber branch: a warm cache does not take the network out of the story.</p></div>
<h2 id="bootstrap"><span class="n" aria-hidden="true">04</span>Bootstrap, fallback and failure modes</h2>
<div class="scene" id="s21"><div class="inner">
<img src="__M21__" alt="Six machine parts arranged in a ring around an empty centre; one of them is a quarter-scale replica of the large way-station machine, representing the mirror's own image passing through the mirror.">
<div class="s21-ring" id="ring"></div>
<div class="cap">The dependency ring: one of the orbiting parts is the mirror itself - its own image is served through the thing it is.</div>
</div></div>
<p>Four circular dependencies live in this design; each got a different answer.</p>
<h3 id="loop1">Loop 1 - the mirror's own image comes through the mirror</h3>
<p>zot runs as a container whose image lives on a registry zot mirrors. When the hosting
node boots, it asks the mirror - which is not running, because the node is booting. The
loop breaks on the documented fallback: mirror endpoints exhaust, containerd falls through
to the origin. Keeping <code>skipFallback</code> unset is the load-bearing choice.</p>
<div class="note"><span class="k">FIELD NOTE &middot; LAB, 2026-06</span><br>Exercised by
accident: a config change shipped with <code>preserveDigest</code> enabled but without its
mandatory partner <code>http.compat</code> - a pairing the registry refuses to start
without. The mirror crashlooped; the fleet fell through to upstream and nothing
user-visible broke. Since then, every config change runs the registry's own
<code>verify</code> in a throwaway pod before merging.</div>
<h3 id="loop2">Loop 2 - the recovery tooling ignores the mirror</h3>
<p>The fleet's rescue tooling does not pull through the mirror: its image is cached on the
operations host, outside the cluster, so it remains available while zot or its cluster is
down.</p>
<h3 id="loop3">Loop 3 - authentication cannot flip everywhere at once</h3>
<p>Enforcing auth needs every node to carry credentials that only apply after a reboot, so
anonymous read must survive until the last node is proven - and the proof must be designed
not to lie. Section 07 covers it.</p>
<h3 id="loop4">Loop 4 - the mirror's host is also its customer</h3>
<p>The node hosting the mirror boots its own workloads through it, including the tunnel
serving this page - so restarting that node carries a blast radius beyond the mirror, and
its reboot runbook lists every public-facing dependant and the order they return.</p>
<div class="diagram" role="region" tabindex="0" aria-label="Bootstrap paths diagram"><span class="k">Diagram D &middot; two bootstrap worlds</span>
<svg viewBox="0 0 860 300" role="img" aria-labelledby="dd-t dd-d">
<title id="dd-t">Connected fail-open bootstrap versus disconnected preseeded bootstrap, as separate paths</title>
<desc id="dd-d">Left: the connected lab path - node boots, mirror miss, implicit fallback to origin, mirror comes up afterwards. Right: the disconnected path - no origin exists; the registry image and release set must be preseeded onto the host or imported from disk before anything else can start.</desc>
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
<text x="215" y="28" fill="#3fbaf5">CONNECTED LAB - FAIL-OPEN (observed)</text>
<rect x="60" y="50" width="310" height="38" rx="8" fill="none" stroke="#3fbaf5"/><text x="215" y="74" fill="#dfe6f0">node boots, asks mirror: down</text>
<rect x="60" y="112" width="310" height="38" rx="8" fill="none" stroke="#e879f9"/><text x="215" y="136" fill="#dfe6f0">implicit fallback -&gt; origin serves boot images</text>
<rect x="60" y="174" width="310" height="38" rx="8" fill="none" stroke="#3fbaf5"/><text x="215" y="198" fill="#dfe6f0">mirror starts (its image came via fallback)</text>
<rect x="60" y="236" width="310" height="38" rx="8" fill="none" stroke="#7fe0a7"/><text x="215" y="260" fill="#dfe6f0">estate converges back onto the mirror</text>
<line x1="215" y1="88" x2="215" y2="108" stroke="#98a2b4"/><line x1="215" y1="150" x2="215" y2="170" stroke="#98a2b4"/><line x1="215" y1="212" x2="215" y2="232" stroke="#98a2b4"/>
<text x="645" y="28" fill="#e8b44a">DISCONNECTED SITE - PRESEEDED (design, not deployed here)</text>
<rect x="490" y="50" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="74" fill="#dfe6f0">no origin exists - fallback is not a plan</text>
<rect x="490" y="112" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="136" fill="#dfe6f0">registry image preseeded / imported from disk</text>
<rect x="490" y="174" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="198" fill="#dfe6f0">release set, trust roots, creds staged locally</text>
<rect x="490" y="236" width="310" height="38" rx="8" fill="none" stroke="#7fe0a7"/><text x="645" y="260" fill="#dfe6f0">site serves itself; syncs on a schedule</text>
<line x1="645" y1="88" x2="645" y2="108" stroke="#98a2b4"/><line x1="645" y1="150" x2="645" y2="170" stroke="#98a2b4"/><line x1="645" y1="212" x2="645" y2="232" stroke="#98a2b4"/>
<line x1="430" y1="40" x2="430" y2="285" stroke="#1c2434"/>
</g>
</svg>
<p class="legend">left = this lab, observed &middot; right = the airgap translation, a design requirement not a deployed claim</p>
<p class="why">Loop 1 assumes an upstream to fall through to; the airgap deletes that
assumption and forces Loop 2's discipline onto the mirror itself.</p></div>
<p><strong>Fallback is a policy decision.</strong> This lab keeps it on for bootstrap
survivability and availability through mirror outages; the cost is that pulls can bypass
the mirror unnoticed and land on origin rate limits. A regulated or disconnected estate
makes the opposite call - <code>skipFallback: true</code>, egress restriction, preseeded
release stock, a tested recovery path - because there the bypass is the failure mode.
Failure classes differ in symptom, path and consequence:</p>
<div class="tablewrap" role="region" tabindex="0" aria-label="Failure mode matrix"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Failure modes - symptom, path, fallback and consequence</caption>
<tr><th scope="col">Failure</th><th scope="col">User-visible symptom</th><th scope="col">Request path</th><th scope="col">Fallback?</th><th scope="col">Detection</th><th scope="col">Security consequence</th></tr>
<tr><td>Mirror pod down</td><td>public pulls may continue transparently; private images, origin limits, DNS or TLS issues can slow or fail them</td><td>node -&gt; origin</td><td>after the mirror endpoint fails; elapsed time differs by failure class (refused vs DNS vs TLS vs blackhole)</td><td>mirror probes red; origin egress rises</td><td>policy/audit bypass while down</td></tr>
<tr><td>Mirror up, sync wedged</td><td>pulls hang up to sync timeout</td><td>node -&gt; mirror (blocked)</td><td>only after timeout</td><td>manifest probe stalls while /v2/ still answers</td><td>availability, not integrity</td></tr>
<tr><td>Upstream 429</td><td>miss/tag pulls fail or crawl</td><td>mirror -&gt; origin refused</td><td>fallback also reaches the origin and may hit a pull or abuse limit - the quota bucket depends on the node's identity and source IP, not necessarily zot's</td><td>zot logs; retry storms amplify</td><td>self-inflicted denial of service via retries</td></tr>
<tr><td>Upstream down, digest cached</td><td>none expected</td><td>mirror serves locally</td><td>not needed</td><td>-</td><td>none (source-verified short-circuit; disconnected drill still open)</td></tr>
<tr><td>Upstream down, tag cached</td><td>reported on v2.1.17: failure or waiting until timeout, with content possibly served after it</td><td>mirror revalidates the tag upstream first</td><td>origin also down</td><td>the test matrix in section 08</td><td>availability surprise for anyone assuming cached means offline-safe (upstream-reported; not yet reproduced here)</td></tr>
<tr><td>PVC full</td><td>new pulls fail, cached OK-ish</td><td>mirror 5xx on writes</td><td>yes for missing content</td><td>capacity metrics (proposed)</td><td>availability; GC pressure</td></tr>
<tr><td>Digest entry GC-evicted</td><td>silent re-fetch on next pull</td><td>mirror -&gt; origin re-sync</td><td>n/a</td><td>upstream egress for "cached" content</td><td>rate-limit exposure returns (section 06)</td></tr>
<tr><td>Node auth wrong (post-flip)</td><td>ImagePullBackOff</td><td>mirror 401; fallback unreliable on 401</td><td>unreliable</td><td>the canary gate (section 07)</td><td>the outage the gate exists to prevent</td></tr>
</table></div>
<h2 id="digests"><span class="n" aria-hidden="true">05</span>Digests, manifests and the supply chain</h2>
<p>Half the mistakes on this page came from conflating these objects:</p>
<ul>
<li>A <strong>tag</strong> is a mutable pointer; zero or many tags can reference one
manifest.</li>
<li>A <strong>manifest digest</strong> identifies exact manifest bytes; a multi-platform
<strong>index</strong>, each platform manifest, and every layer and config blob carry their
own.</li>
<li><code>name:tag@digest</code> is client-side grammar, not wire protocol: the OCI
distribution spec takes a tag <em>or</em> a digest in the URL. Clients resolve the digest
and ignore the tag (Kubernetes documents this), so digest-pinned pulls are immune to tag
moves. This page once mislearned the combined form as a cache-miss bug; retesting on
v2.1.17 showed the historical zot rejection no longer reproduces.</li>
<li>Digest pinning provides <strong>content integrity</strong> - not provenance, not
authorisation, not availability, and not freedom from network fetches on a cold store.</li>
</ul>
<p>For a mirror, digests carry one sharp operational rule: zot converts Docker-schema
manifests to OCI by default, and conversion changes the digest - breaking every digest pin
and signature downstream. The pairing that prevents it (<code>preserveDigest: true</code>
per upstream, with <code>http.compat: ["docker2s2"]</code>) is mandatory in both
directions: the registry refuses to start with one and not the other (the Loop 1 field
note is this rule being learned). This deployment runs both on all five upstreams.</p>
<p>Digest preservation and artefact completeness are different properties. The docs tie
<code>preserveDigest</code>/<code>compat</code> to keeping mirrored manifest bytes and
media types - and therefore signature <em>validity</em> - aligned with upstream. Whether
signature, SBOM and attestation objects <em>arrive</em> is separate: OCI 1.1 referrers
ride the <code>subject</code> field and their own API, legacy Cosign signatures ride
tag-schema conventions, and both depend on origin support and this version's sync
behaviour. This deployment has not verified a complete referrer graph for any origin and
performs no cryptographic signature verification; disconnected verification would also
need locally held trust material. Both gaps sit in the open register. Production estates
promote releases by digest and decide per policy whether referrers travel too; this lab
runs only the digest half.</p>
<h2 id="retention"><span class="n" aria-hidden="true">06</span>Retention, GC and the digest-only trap</h2>
<div class="scene" id="s23"><div class="inner">
<img src="__M23__" alt="A stuck intake pod ringed in magenta with three glowing cargo bricks queued behind it - cached content waiting on a jammed process.">
<div class="s23-pulse" id="pulse"></div>
<div class="cap">Cached bricks queued behind a jammed intake: availability problems here are quiet, not loud.</div>
</div></div>
<p>This section corrects the largest error in this page's first edition, which said "no
retention config means keep everything". True for tags; false for everything else - and
"everything else" includes the content this mirror exists to hold. The semantics, from
upstream documentation and the deployed version's source:</p>
<ul>
<li>With no retention configuration, <strong>all tags are retained</strong> and
<strong>all untagged manifests are deleted</strong> by GC (unless referenced by an index
or artefact). Timing is two-stage: the manifest ages past <code>gcDelay</code> (1h here),
then deletion happens at the next sweep (<code>gcInterval</code>, 24h here) - the survival
window depends on where creation falls relative to that sweep, not a fixed one-hour fuse.</li>
<li><strong>Digest-only pull-through entries are stored as untagged manifests.</strong>
That is the upstream fix's own problem statement: pull an image through the mirror by
digest, and the cache entry has no tag to protect it.</li>
</ul>
<p>Combined with section 04's revalidation behaviour: <em>tag</em> pulls are cached but
contact upstream anyway; <em>digest</em> pulls serve locally but their entries are
GC-eligible between sweeps - a digest-pinned fleet caches exactly the entry class GC may
delete. On v2.1.17 this mirror is a rate-limit shield and a latency win, <strong>not yet a
disconnection shelf</strong>; the first edition implied otherwise and was wrong.</p>
<div class="diagram" role="region" tabindex="0" aria-label="Storage and retention diagram"><span class="k">Diagram C &middot; what keeps an object alive</span>
<svg viewBox="0 0 860 360" role="img" aria-labelledby="dc-t dc-d">
<title id="dc-t">The OCI object graph and which references protect content from garbage collection</title>
<desc id="dc-d">A tag points to an index; the index references platform manifests; manifests reference layers and a config blob. Referrers attach to a manifest via a subject field. A separate digest-only manifest sits with no tag pointing at it; it is eligible for garbage collection after the delay on the deployed version. Retention rules that keep tags do not protect the untagged manifest.</desc>
<defs><marker id="dc-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
<marker id="dc-b" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#98a2b4"/></marker></defs>
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
<rect x="30" y="30" width="150" height="40" rx="8" fill="none" stroke="#7fe0a7"/><text x="105" y="55" fill="#dfe6f0">tag: v1.2</text>
<rect x="240" y="30" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="325" y="49" fill="#dfe6f0">index (multi-arch)</text><text x="325" y="64" fill="#98a2b4">own digest</text>
<rect x="470" y="10" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="555" y="35" fill="#dfe6f0">manifest amd64</text>
<rect x="470" y="62" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="555" y="87" fill="#dfe6f0">manifest arm64</text>
<rect x="700" y="10" width="140" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="770" y="35" fill="#dfe6f0">layers + config</text>
<rect x="700" y="62" width="140" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="770" y="87" fill="#dfe6f0">layers + config</text>
<rect x="470" y="130" width="170" height="40" rx="8" fill="none" stroke="#e879f9" stroke-dasharray="4 3"/><text x="555" y="149" fill="#dfe6f0">referrer (sig/SBOM)</text><text x="555" y="164" fill="#e879f9">subject -&gt; manifest</text>
<line x1="180" y1="50" x2="235" y2="50" stroke="#7fe0a7" marker-end="url(#dc-a)"/>
<line x1="410" y1="42" x2="465" y2="30" stroke="#3fbaf5" marker-end="url(#dc-a)"/>
<line x1="410" y1="58" x2="465" y2="80" stroke="#3fbaf5" marker-end="url(#dc-a)"/>
<line x1="640" y1="30" x2="695" y2="30" stroke="#98a2b4" marker-end="url(#dc-b)"/>
<line x1="640" y1="82" x2="695" y2="82" stroke="#98a2b4" marker-end="url(#dc-b)"/>
<line x1="555" y1="130" x2="555" y2="106" stroke="#e879f9" stroke-dasharray="4 3" marker-end="url(#dc-a)"/>
<text x="105" y="120" fill="#7fe0a7">a tag is a keep-alive</text>
<rect x="30" y="215" width="280" height="52" rx="8" fill="none" stroke="#e8b44a"/><text x="170" y="236" fill="#dfe6f0">digest-only cached manifest</text><text x="170" y="253" fill="#e8b44a">NO tag references it</text>
<rect x="380" y="215" width="220" height="52" rx="8" fill="none" stroke="#ff8fa3"/><text x="490" y="236" fill="#dfe6f0">GC after gcDelay (1h here)</text><text x="490" y="253" fill="#ff8fa3">deleteUntagged default: true</text>
<line x1="310" y1="241" x2="375" y2="241" stroke="#ff8fa3" marker-end="url(#dc-a)"/>
<text x="430" y="310" fill="#98a2b4">keepTags protects TAGS in its repo. On v2.1.17/18 the amber box has one blanket lever: deleteUntagged:false,</text>
<text x="430" y="330" fill="#98a2b4">which keeps ALL untagged manifests (capacity trade). Selective pull-aware keepUntagged ships in v2.1.19.</text>
</g>
</svg>
<p class="legend">green = protected by a tag &middot; amber/red = the eviction path &middot; dashed magenta = referrers (separate lifecycle)</p>
<p class="why">GC evaluates reference reachability: an untagged digest-only manifest can be
deleted even while its blobs remain present, and node-local image stores can mask the
eviction for days.</p></div>
<p>Retention policy semantics - two nearby systems use opposite matching rules:</p>
<ul>
<li>Retention policies match per repository, <strong>first match wins</strong> - order the
list most-specific-first. A repository matching no policy keeps its tags.</li>
<li>Within a matched repository, one <code>keepTags</code> rule inverts the default:
non-matching tags in <em>that repository</em> become deletable. The inversion is scoped to
the repository, not global - the first edition of this page overstated it.</li>
<li>Untagged manifests are governed separately (<code>deleteUntagged</code>, default true),
and on the deployed version no retention rule can protect them. Pull-aware
<code>keepUntagged</code> exists upstream from v2.1.19.</li>
<li>Authorisation policies (section 07) use <strong>longest match</strong>, not first
match. Two adjacent config blocks, two opposite precedence rules. Label which one you are
reasoning about.</li>
</ul>
<p><strong>Mitigations, ranked:</strong> upgrade to v2.1.19+ for pull-aware
<code>keepUntagged</code> (the designed fix, schema to be validated against the actual
binary before rollout); until then, widen <code>gcDelay</code>/<code>retention.delay</code>
(the upstream maintainer's interim suggestion), or set <code>deleteUntagged: false</code>,
which protects every untagged manifest at the cost of unbounded cache growth. Disabling GC
entirely swaps eviction for storage exhaustion with no reclaim path.</p>
<div class="note warn"><span class="k">SAFE VALIDATION - PROPOSED, NOT YET RUN</span><br>
The eviction claim is documented upstream and consistent with this config; it has not been
reproduced here. The reproduction, in a <em>disposable</em> instance only: record version
and sanitised config; pull by digest; confirm the stored manifest is untagged; shorten GC
timers; observe the manifest across a sweep; then block upstream and re-pull from a clean
runtime store, recording serve vs re-fetch. Two upstream warnings: the retention
verification tool executes orphan-blob GC for real even in dry-run, and local storage
requires the registry stopped. Never point retention experiments at live storage.</div>
<h2 id="auth"><span class="n" aria-hidden="true">07</span>The authentication migration</h2>
<div class="scene" id="s22"><div class="inner">
<img src="__M22__" alt="Four parts left to right: a waiting keyed cartridge, a key wedge, a tall gate frame, and a cartridge beyond the gate with its keyway lit - per-node credentials proven at a checkpoint.">
<div class="s22-glow" style="left:16%;top:30%" data-band="0"></div>
<div class="s22-glow" style="left:44%;top:28%" data-band="1"></div>
<div class="s22-glow" style="left:70%;top:26%" data-band="2"></div>
<div class="cap">The proving gate: a node counts as migrated when an authenticated pull succeeds where an anonymous one cannot.</div>
</div></div>
<p>Target state: the mirror refuses anonymous pulls. Current state: anonymous read on,
per-node credentials staged inert in machine configs until each reboot. The sequencing
problem has one central hazard: <strong>while anonymous read is on, an ordinary pull says
nothing about auth</strong> - a node whose credentials never applied is served as an
anonymous reader and false-passes the gate. The canary must live in a repository that
denies anonymous read, so success can only mean an authenticated pull.</p>
<p>The design is upstream-supported: authorisation resolves per-repository policies by
<strong>longest match</strong> (<code>**</code> is the default for anything unmatched), and
maintainer guidance confirms anonymous and authenticated access evaluate independently. So
a <code>canary/**</code> entry granting named identities read, with no
<code>anonymousPolicy</code>, denies anonymous on that path while the glob keeps the rest
of the shelf open. The staged policy:</p>
<pre>"accessControl": {
"repositories": {
"**": {
"anonymousPolicy": ["read"],
"policies": [
{ "users": ["zot-push"], "actions": ["read", "create", "update"] },
{ "users": ["zot-pull"], "actions": ["read"] }
]
},
"canary/**": {
"policies": [
{ "users": ["zot-pull", "zot-push"], "actions": ["read"] }
]
}
}
}</pre>
<div class="note"><span class="k">CLIENT CAVEAT &middot; DOCUMENTED</span><br>Mixed
anonymous/authenticated policies trigger a Docker-client-specific workaround present in
this exact version: <code>/v2/</code> returns 401 to Docker user agents so the Docker CLI
sends credentials, meaning anonymous <em>docker</em> users must log in even for anonymous
repos. Podman and containerd are unaffected. This estate pulls with containerd, podman and
buildah, so the caveat is noted rather than felt.</div>
<p>The gate, per node, and what each step proves:</p>
<ol>
<li>Stage credentials in the node's machine config (inert; documented as requiring reboot).</li>
<li>Reboot the node at a planned window.</li>
<li><strong>Identity check:</strong> pull the protected canary through the node's own
runtime. Success = authenticated (anonymous cannot); 401 = back to step 1. The expected
split - 401 for missing/wrong credentials, 403 for a valid identity lacking the action -
sits in the open register, since the docs state the 403 case only for OIDC identities.</li>
<li><strong>Interception check, separately:</strong> the canary is a direct mirror
reference, so it says nothing about whether <code>docker.io/...</code> references route
through the mirror. That needs an original-upstream reference pulled on a node with a cold
runtime store, correlated with the mirror's request log - node-local content would satisfy
the pull without any network and fake a pass.</li>
<li>Anonymous read comes off the glob only when every node passes both checks. Rollback is
the previous config commit, named before the flip.</li>
</ol>
<div class="diagram" role="region" tabindex="0" aria-label="Authentication migration diagram"><span class="k">Diagram E &middot; the migration ladder</span>
<svg viewBox="0 0 860 210" role="img" aria-labelledby="de-t de-d">
<title id="de-t">The authentication migration ladder with its per-node proof gate and rollback point</title>
<desc id="de-d">Five stages: anonymous baseline, credentials staged inert, node reboot, the two-part proof - authenticated canary pull plus logged mirror interception - and the final anonymous-off flip with a named rollback commit. A failing node loops from the proof back to staging.</desc>
<defs><marker id="de-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
<marker id="de-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker></defs>
<g font-family="ui-monospace,Menlo,monospace" font-size="11" text-anchor="middle">
<rect x="10" y="70" width="140" height="48" rx="8" fill="none" stroke="#98a2b4"/><text x="80" y="90" fill="#dfe6f0">anon read ON</text><text x="80" y="106" fill="#98a2b4">baseline (today)</text>
<rect x="180" y="70" width="140" height="48" rx="8" fill="none" stroke="#3fbaf5"/><text x="250" y="90" fill="#dfe6f0">creds staged</text><text x="250" y="106" fill="#98a2b4">inert in config</text>
<rect x="350" y="70" width="140" height="48" rx="8" fill="none" stroke="#3fbaf5"/><text x="420" y="90" fill="#dfe6f0">node reboots</text><text x="420" y="106" fill="#98a2b4">auth goes live</text>
<rect x="520" y="70" width="160" height="48" rx="8" fill="none" stroke="#e879f9"/><text x="600" y="86" fill="#dfe6f0">two-part proof</text><text x="600" y="100" fill="#e879f9">canary + logged</text><text x="600" y="113" fill="#e879f9">interception</text>
<rect x="710" y="70" width="140" height="48" rx="8" fill="none" stroke="#7fe0a7"/><text x="780" y="90" fill="#dfe6f0">anon OFF</text><text x="780" y="106" fill="#98a2b4">rollback = prior commit</text>
<line x1="150" y1="94" x2="175" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
<line x1="320" y1="94" x2="345" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
<line x1="490" y1="94" x2="515" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
<line x1="680" y1="94" x2="705" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
<path d="M600 118 Q 600 175 480 175 L 320 175 Q 250 175 250 122" fill="none" stroke="#e879f9" stroke-dasharray="4 4" marker-end="url(#de-m)"/>
<text x="440" y="196" fill="#e879f9">401: node loops back - it does not count</text>
</g>
</svg>
<p class="legend">the flip is per-fleet; the proof is per-node - one unproven node under a fleet-wide flip is an outage wearing a green tick</p>
<p class="why">Identity and interception fail independently; each has a false-pass mode
the other cannot detect.</p></div>
<p>Five trust domains never share material: node pull credentials (machine config,
reboot-bound), the push credential (used by the operator-run build host for registry
logins, not embedded in CI configuration), the mirror's upstream credentials (one mounted
secret, pod-restart-bound), TLS trust, and human admin access (SSO in front of the UI).
The push credential carries this page's oldest incident: in July 2026 its bcrypt hash and
plaintext drifted apart in the secrets manager, rotation became impossible, and pushes
stayed dead for twelve days - three applications shipped the day the rebuilt flow landed.
Today the hash and plaintext are separate entries in the same secrets manager, rotated as
a pair by a human; the registry consumes only the derived htpasswd file, each plaintext
reaches only its consumer, and the sync identity is read-only so automation cannot
half-rotate the pair. Residual cost, stated: one secrets-manager project holds both
halves, so its compromise yields verifier and credential together - versioned,
consumer-scoped secret objects remain a possible refinement.</p>
<h2 id="observability"><span class="n" aria-hidden="true">08</span>Observability that means something</h2>
<p>The worst incident in this mirror's life: an in-flight sync wedged, every pull from
that upstream hung, and the health endpoint returned 200 throughout:</p>
<div class="pane-k">Replay &middot; observed 2026-07-07 &middot; the stalled sync</div>
<pre class="pane">"image already demanded, waiting on channel"
$ kubectl -n zot rollout restart deploy zot <span class="cm">&lt;- recovery action (not a root cause)</span>
<span class="cm"># probe an actual manifest, not the process's opinion of itself:</span>
$ curl -sfS --max-time 10 -o /dev/null -w '%{http_code}\n' \
https://zot.bztmon.org/v2/library/busybox/manifests/latest \
-H 'Accept: application/vnd.oci.image.index.v1+json'
<span class="good">200</span> <span class="cm">&lt;- THIS is "the mirror works"</span></pre>
<p>The deployed source refines that story: the log line is coalescing by design -
concurrent requests join the first sync, which runs on a detached background context with
a three-hour default timeout and survives client disconnects. Waiters blocking on a
stalled sync until that timeout matches what we saw; the restart was recovery, and the
root cause was never isolated. Two knobs this config leaves unset - sync
<code>maxRetries</code> (disabled by default upstream) and a tighter
<code>syncTimeout</code> - are the first change on recurrence.</p>
<p>From the running version's source: <code>/livez</code>, <code>/readyz</code> and
<code>/startupz</code> exist as health endpoints (absent from this version's docs),
alongside the spec's <code>/v2/</code>. The Prometheus metrics extension exists upstream -
series names include <code>zot_http_requests_total</code>,
<code>zot_repo_storage_bytes</code>, <code>zot_repo_downloads_total</code> and
<code>zot_storage_lock_latency_seconds</code> - and is <strong>not enabled here</strong>;
wiring it, with scrape auth, is on the open list.</p>
<p>The layered checks, one question each:</p>
<div class="tablewrap" role="region" tabindex="0" aria-label="Layered operational checks"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Layered checks - one question each</caption>
<tr><th scope="col">Check</th><th scope="col">Question it answers</th><th scope="col">Status here</th></tr>
<tr><td><code>/livez</code> / <code>/readyz</code></td><td>is the process up / initialised</td><td><span class="st ok">available</span></td></tr>
<tr><td>authenticated manifest GET of a local-only canary</td><td>can the right identity read real content from local storage</td><td><span class="st prop">proposed</span></td></tr>
<tr><td>anonymous GET of the canary expecting 401</td><td>is the protection actually protecting</td><td><span class="st prop">proposed</span></td></tr>
<tr><td>original-reference pull on a cold node + mirror log line</td><td>is the runtime actually routed through the mirror</td><td><span class="st prop">proposed</span></td></tr>
<tr><td>digest pull with upstream blocked (disposable env)</td><td>does "cached" mean "served locally"</td><td><span class="st open">open - section 06</span></td></tr>
<tr><td>PVC usage + growth, GC activity, sync latency</td><td>when does capacity or eviction become the story</td><td><span class="st prop">needs metrics ext</span></td></tr>
<tr><td>certificate expiry, credential age</td><td>what breaks on a schedule</td><td><span class="st prop">proposed</span></td></tr>
</table></div>
<p>The caching test matrix (digest rows: source-verified and consistent with behaviour
here; disconnected rows: not yet run in this lab):</p>
<div class="tablewrap" role="region" tabindex="0" aria-label="Cache behaviour test matrix"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Cache behaviour - each row is an experiment</caption>
<tr><th scope="col">Scenario</th><th scope="col">Expected on v2.1.17</th><th scope="col">What proves it</th></tr>
<tr><td>cached tag, upstream reachable</td><td>served; upstream contacted anyway (revalidation)</td><td>zot log shows upstream request; latency includes round-trip</td></tr>
<tr><td>cached tag, upstream unreachable</td><td>fails or stalls until timeout, then local fallback path</td><td>pull timing vs sync timeout; error text</td></tr>
<tr><td>cached digest, upstream reachable</td><td>served locally, no upstream contact</td><td>absence of upstream request in zot logs during the pull</td></tr>
<tr><td>cached digest, upstream unreachable</td><td>served locally - IF the entry survived GC</td><td>the section-06 disposable-instance drill</td></tr>
<tr><td>cold node, warm mirror</td><td>mirror serves; node store fills</td><td>mirror log + no origin egress</td></tr>
<tr><td>warm node, evicted mirror entry</td><td>pull succeeds from node store - masking the eviction</td><td>this is the false-pass: only mirror-side inspection reveals it</td></tr>
<tr><td>multi-arch index + child manifests</td><td>index and per-platform manifests are separate cache entries</td><td>per-digest existence checks on the mirror</td></tr>
</table></div>
<h2 id="production"><span class="n" aria-hidden="true">09</span>The production translation</h2>
<div class="scene" id="s24"><div class="inner">
<img src="__M24__" alt="A monolithic vault with a wall of glowing bricks visible through its door, attended by four smaller self-sufficient way-stations - a central release source and site-local mirrors.">
<div class="cap">A central curated source and site-local registries: each site holds its own shelf, so a cut WAN idles nothing.</div>
</div></div>
<p>The first edition of this section asserted what "production" does; production chooses
from patterns against requirements:</p>
<ul>
<li><strong>Single registry, accepted failure domain</strong> - this lab's shape.
Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or a paused
deploy window (fallback off), with the cost accepted in writing.</li>
<li><strong>Sync-based HA</strong> - documented active/standby or active/active pairs of
<em>independent</em> instances, each with its own storage, mirroring each other behind a
load balancer. The caveat: the synchronisation window between polls bounds what failover
can lose.</li>
<li><strong>Scale-out clustering</strong> - instances shard repositories by hash and proxy
to the owner, classically over shared S3-compatible storage. Horizontal scale for load;
the shared storage becomes the availability story - a different property from the HA pair.</li>
<li><strong>Edge estates</strong> - a central source promotes releases by digest; each
site runs a local registry holding its release set on a schedule, so a severed WAN idles
nothing at pull time. A disconnection claim is valid only when content, required
referrers, trust roots, credentials and the registry's own bootstrap image are already
local (diagram D's right-hand column, per site).</li>
</ul>
<p>Whatever the topology: recovery time and point objectives get numbers before an
incident supplies them; backups are restore-tested copies off the failure domain (array
snapshots and RAID protect against disk loss, not against the array, the site or operator
error); cache capacity is planned from the section 06 retention policy; and the section 04
circular dependencies get drawn per site, because each exists at every scale - only the
cost of ignoring them changes.</p>
<h2 id="verified"><span class="n" aria-hidden="true">10</span>Verified, open, and where the claims come from</h2>
<div class="scene" id="s25"><div class="inner">
<img src="__M25__" alt="A wide flat node slab in an otherwise empty void; a single luminous brick hovers midway in its descent toward the slab - one small transfer of bytes as proof of life.">
<div class="cap">Proof of life is one authenticated byte transfer through the real path - not a status page.</div>
</div></div>
<p>When a node powers on, the fleet's admission check performs an authenticated manifest
pull through the node's own runtime before the node counts as a member: a process-health
response validates neither routing nor authorisation. Most of this page condenses into
that one command.</p>
<div class="tablewrap" role="region" tabindex="0" aria-label="Architecture decision table"><table>
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Architecture decisions</caption>
<tr><th scope="col">Decision</th><th scope="col">Chosen</th><th scope="col">Why</th><th scope="col">Trade-off</th><th scope="col">Rollback / alternative</th><th scope="col">Status</th></tr>
<tr><td>Origin fallback</td><td>ON (default kept)</td><td>bootstrap survivability, Loop 1</td><td>silent mirror bypass possible</td><td>skipFallback:true + preseed (airgap shape)</td><td><span class="st ok">observed</span></td></tr>
<tr><td>Sync mode</td><td>onDemand, all upstreams</td><td>cache follows real usage; Hub-safe</td><td>tag pulls revalidate upstream; digest entries untagged</td><td>polled sync for a curated release set</td><td><span class="st ok">observed</span></td></tr>
<tr><td>Digest preservation</td><td>preserveDigest + docker2s2, x5</td><td>digest pins + signatures survive mirroring</td><td>mandatory config pairing (crashloop scar)</td><td>none - non-negotiable for digest-pinned estates</td><td><span class="st ok">observed</span></td></tr>
<tr><td>Namespace layout</td><td>flat (no destinations)</td><td>simplicity at build time</td><td>ordered-trial upstream selection; theoretical collisions</td><td>per-origin destinations + overridePath</td><td><span class="st prop">redesign proposed</span></td></tr>
<tr><td>Retention</td><td>none configured</td><td>predates understanding the untagged rule</td><td>digest-only entries evict within hours</td><td>v2.1.19+ keepUntagged; interim wider gcDelay</td><td><span class="st open">correction owed</span></td></tr>
<tr><td>Auth</td><td>anonymous read until per-node proof</td><td>containerd 401-fallback unreliability makes big-bang flips dangerous</td><td>window with no pull auth</td><td>staged flip w/ canary gate + named rollback</td><td><span class="st open">in flight</span></td></tr>
<tr><td>Metrics</td><td>not enabled</td><td>minimal first deployment</td><td>capacity/eviction invisible</td><td>metrics extension + scrape auth</td><td><span class="st prop">proposed</span></td></tr>
</table></div>
<h3 id="open">Open verification register</h3>
<ol>
<li>Digest-entry eviction reproduction in a disposable instance (section 06 procedure).</li>
<li>Canary repo negative test: anonymous 401 on <code>canary/**</code> while anonymous
read elsewhere still succeeds; observed 401-vs-403 boundaries recorded.</li>
<li>Mirror interception proof: original-reference pull on a cold node correlated with the
mirror's request log.</li>
<li>Disconnected-serve drill: cached digest pull with upstream blocked, disposable
environment first.</li>
<li>Upgrade evaluation: v2.1.19/v2.1.20 for <code>keepUntagged</code>, with the retention
config written and reviewed before the upgrade, not after.</li>
<li>Namespace redesign migration plan (per-origin destinations + overridePath), including
cache re-warm cost.</li>
</ol>
<h3 id="refs">Verified against</h3>
<ul class="refs">
<li>zot documentation (versioned): <a href="https://zotregistry.dev/v2.1.18/articles/retention/">retention</a> &middot; <a href="https://zotregistry.dev/v2.1.18/articles/storage/">storage</a> &middot; <a href="https://zotregistry.dev/v2.1.18/articles/mirroring/">mirroring</a> &middot; <a href="https://zotregistry.dev/v2.1.18/articles/authn-authz/">authn/authz</a> &middot; <a href="https://zotregistry.dev/v2.1.18/articles/high-availability/">high availability</a> &middot; <a href="https://zotregistry.dev/v2.1.18/articles/scaleout/">scale-out</a></li>
<li>zot upstream issues/PRs: <a href="https://github.com/project-zot/zot/issues/4148">4148 digest-only eviction</a> &middot; <a href="https://github.com/project-zot/zot/pull/4191">4191 keepUntagged (v2.1.19)</a> &middot; <a href="https://github.com/project-zot/zot/issues/4128">4128 tag revalidation</a> &middot; <a href="https://github.com/project-zot/zot/issues/4187">4187 containerd ns param</a> &middot; <a href="https://github.com/project-zot/zot/issues/2928">2928 mixed anonymous auth</a> &middot; <a href="https://github.com/project-zot/zot/issues/2505">2505 demand coalescing</a></li>
<li>Talos: <a href="https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/images-container-runtime/pull-through-cache">pull-through registry cache (v1.13)</a> &middot; containerd: <a href="https://github.com/containerd/containerd/blob/main/docs/hosts.md">registry hosts and the ns parameter</a></li>
<li>OCI: <a href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md">distribution spec</a> &middot; <a href="https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/">image/distribution 1.1 (referrers)</a> &middot; signing: <a href="https://docs.sigstore.dev/cosign/verifying/verify/">cosign verify</a></li>
<li>Docker Hub <a href="https://docs.docker.com/docker-hub/usage/">usage and limits</a> &middot; buildah <a href="https://github.com/containers/buildah/blob/main/docs/buildah-build.1.md">build --retry semantics</a></li>
</ul>
<footer class="outro">
<p>&copy; 2026 Jonathon Wright &middot; <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC BY-NC-SA 4.0</a>
&middot; hand-built static, shipped via GitOps &middot; <a href="https://www.bztmon.com/">bztmon.com</a></p>
</footer>
</div>
<script src="__MIRROR_JS__"></script>
Reference in New Issue View Git Blame Copy Permalink