jwright
f6616a5863
The three pages had each grown their own tokens, type scale and alignment, so the openings looked like three different sites. The document shell now owns the masthead family - tokens, centred masthead, eyebrow, title, lede, fact strip, scroll cue and the index component - and the pages carry only their content. The Mirror gains the section index the arc has, drops its duplicate technical footer, and its title now uses the same magenta accent instead of a one-off gradient.
834 lines
67 KiB
HTML
834 lines
67 KiB
HTML
<style>
|
|
*{box-sizing:border-box}
|
|
html{scroll-behavior:smooth}
|
|
body{margin:0;background:var(--void);color:var(--ink);
|
|
font-family:var(--sans);font-size:1.05rem;line-height:1.65}
|
|
#bar{position:fixed;top:0;left:0;height:2px;width:0;background:linear-gradient(90deg,var(--cyan),var(--magenta));z-index:9}
|
|
/* full-bleed via named grid lines - no viewport units, so the vertical scrollbar
|
|
can never make the document wider than the visible viewport */
|
|
.layout{display:grid;
|
|
grid-template-columns:[full-start] minmax(1.25rem,1fr) [main-start] min(46rem,calc(100% - 2.5rem)) [main-end] minmax(1.25rem,1fr) [full-end]}
|
|
.layout>*{grid-column:main}
|
|
.layout>.scene{grid-column:full}
|
|
.eyebrow{font-family:var(--mono);font-size:.78rem;letter-spacing:.22em;color:var(--cyan);text-transform:uppercase}
|
|
h1{font-family:var(--sans);font-weight:750;font-size:clamp(2.8rem,8vw,4.6rem);
|
|
line-height:.98;margin:.5rem 0 1rem;text-wrap:balance;letter-spacing:-.01em}
|
|
.chapnav{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;margin:0 0 2.4rem}
|
|
.chapnav a{color:var(--ink-dim);text-decoration:none;border-bottom:1px solid transparent;
|
|
padding-bottom:2px;transition:color .2s ease,border-color .2s ease}
|
|
.chapnav a:hover,.chapnav a:focus-visible{color:var(--cyan);border-color:var(--cyan)}
|
|
h1 em{font-style:normal;background:linear-gradient(100deg,var(--cyan),var(--magenta));
|
|
-webkit-background-clip:text;background-clip:text;color:transparent;padding-bottom:.12em}
|
|
.standfirst{font-size:clamp(1.15rem,2.4vw,1.5rem);color:var(--ink);font-weight:600;
|
|
margin:.2rem 0 1rem;letter-spacing:-.01em;max-width:34rem}
|
|
h2{font-family:var(--sans);font-weight:700;font-size:1.85rem;letter-spacing:0;
|
|
margin:4.2rem 0 1rem;text-wrap:balance;scroll-margin-top:2.5rem}
|
|
h2 .n{color:var(--cyan);font-family:var(--mono);font-weight:500;font-size:1rem;vertical-align:.35em;margin-right:.65rem;letter-spacing:.1em}
|
|
h3{font-weight:650;font-size:1.28rem;margin:2.2rem 0 .6rem;scroll-margin-top:2.5rem}
|
|
p{margin:.9rem 0}
|
|
code,.mono{font-family:var(--mono);font-size:.92em}
|
|
.scene{margin:3rem 0;overflow:hidden;position:relative}
|
|
.scene .inner{max-width:72rem;margin:0 auto;padding:0 1rem;position:relative}
|
|
.scene img{width:100%;height:auto;display:block;border-radius:.4rem}
|
|
.scene .cap{font-family:var(--mono);font-size:.74rem;color:var(--ink-dim);
|
|
letter-spacing:.1em;text-align:center;margin-top:.7rem;padding:0 1rem}
|
|
[tabindex="0"]:focus-visible{outline:2px solid var(--cyan);outline-offset:3px}
|
|
.note{background:var(--card);border:1px solid var(--line);border-left:3px solid var(--cyan);
|
|
border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
|
|
.note .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan)}
|
|
.note.warn{border-left-color:var(--amber)}
|
|
.note.warn .k{color:var(--amber)}
|
|
.clause{background:linear-gradient(135deg,rgba(232,121,249,.07),transparent 60%);
|
|
border:1px solid var(--line);border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
|
|
.clause .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--magenta)}
|
|
pre{background:#050810;border:1px solid var(--line);border-radius:.5rem;padding:1rem 1.1rem;
|
|
overflow-x:auto;font-family:var(--mono);font-size:.83rem;line-height:1.55;color:#c8d2e0}
|
|
pre.pane{border-left:3px solid var(--cyan)}
|
|
pre .cm{color:#8a95ab}
|
|
pre .good{color:var(--good)}
|
|
pre .bad{color:var(--bad)}
|
|
.pane-k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--ink-dim);
|
|
text-transform:uppercase;margin:1.6rem 0 .4rem}
|
|
.diagram{background:var(--card);border:1px solid var(--line);border-radius:.55rem;
|
|
padding:1.2rem;margin:2rem 0;overflow-x:auto}
|
|
.diagram .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan);display:block;margin-bottom:.8rem}
|
|
.diagram svg{width:100%;height:auto;display:block;min-width:560px}
|
|
.diagram .why{font-size:.9rem;color:var(--ink-dim);margin:.8rem 0 0}
|
|
.diagram .legend{font-family:var(--mono);font-size:.72rem;color:var(--ink-dim);margin:.6rem 0 0}
|
|
table{border-collapse:collapse;width:100%;font-size:.88rem;margin:1.4rem 0}
|
|
.tablewrap{overflow-x:auto;margin:1.4rem 0}
|
|
.tablewrap table{margin:0;min-width:640px}
|
|
th,td{border:1px solid var(--line);padding:.5rem .65rem;text-align:left;vertical-align:top}
|
|
th{font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;text-transform:uppercase;color:var(--cyan);background:var(--void-2)}
|
|
td .st{font-family:var(--mono);font-size:.7rem;letter-spacing:.06em;padding:.05rem .4rem;border-radius:.25rem;border:1px solid var(--line);white-space:nowrap}
|
|
.st.ok{color:var(--good);border-color:rgba(127,224,167,.4)}
|
|
.st.open{color:var(--amber);border-color:rgba(232,180,74,.4)}
|
|
.st.prop{color:var(--magenta);border-color:rgba(232,121,249,.4)}
|
|
.facts{border:1px solid var(--line);border-radius:.55rem;background:var(--card);
|
|
padding:1.1rem 1.3rem;margin:2.2rem 0;font-size:.92rem}
|
|
.facts dl{display:grid;grid-template-columns:max-content 1fr;gap:.35rem 1.2rem;margin:0}
|
|
.facts dt{font-family:var(--mono);font-size:.72rem;letter-spacing:.12em;text-transform:uppercase;color:var(--ink-dim);padding-top:.15rem}
|
|
.facts dd{margin:0}
|
|
ol{padding-left:1.3rem} ol li{margin:.6rem 0}
|
|
ul{padding-left:1.2rem} ul li{margin:.5rem 0}
|
|
em{color:#fff}
|
|
.traps p{margin:1.1rem 0}
|
|
.traps b,strong{color:#fff}
|
|
footer{margin:5rem 0 4rem;border-top:1px solid var(--line);padding-top:1.4rem;
|
|
color:var(--ink-dim);font-size:.88rem;font-family:var(--mono)}
|
|
a{color:var(--cyan)}
|
|
:focus-visible{outline:2px solid var(--cyan);outline-offset:2px}
|
|
.s21-ring{position:absolute;inset:0;pointer-events:none;mix-blend-mode:screen;opacity:.5;
|
|
background:conic-gradient(from 0deg at 50% 50%, transparent 0deg, rgba(63,186,245,.28) 24deg, transparent 60deg)}
|
|
.s22-glow{position:absolute;width:24%;aspect-ratio:1;border-radius:50%;pointer-events:none;
|
|
mix-blend-mode:screen;opacity:0;background:radial-gradient(circle,rgba(63,186,245,.5),transparent 65%)}
|
|
.s23-pulse{position:absolute;left:34%;top:44%;width:26%;aspect-ratio:1;border-radius:50%;pointer-events:none;
|
|
mix-blend-mode:screen;opacity:.25;background:radial-gradient(circle,rgba(232,121,249,.55),transparent 62%)}
|
|
.refs li{margin:.45rem 0;font-size:.92rem}
|
|
.outro{max-width:64ch;margin:0 auto;padding:2vh 24px 14vh;text-align:center;border:0}
|
|
.outro p{font-family:var(--mono);font-size:.7rem;letter-spacing:.12em;color:var(--ink-dim);line-height:1.9;margin:0}
|
|
@media (prefers-reduced-motion: reduce){
|
|
html{scroll-behavior:auto}
|
|
.s21-ring,.s23-pulse{display:none}
|
|
.s22-glow{opacity:.35}
|
|
}
|
|
</style>
|
|
|
|
<div id="bar"></div>
|
|
|
|
<div class="layout">
|
|
|
|
<header class="masthead">
|
|
<nav class="chapnav" aria-label="Site"><a href="/">← The Teaching Lab</a></nav>
|
|
<p class="eyebrow">Chapter 02 · The Mirror · Dossier 005</p>
|
|
<h1>The cache was the easy part.<br><em>Now watch what it quietly throws away.</em></h1>
|
|
<p class="lede">A pull-through registry, taken apart: what the deployed version actually
|
|
guarantees, the retention behaviour that evicts digest-pinned content, and how each lesson
|
|
translates to production. Every claim carries an evidence label - including the corrections
|
|
to my own first draft. Scroll, and the machinery comes apart.</p>
|
|
<dl class="mastfacts">
|
|
<div><dt>Registry</dt><dd>zot v2.1.17</dd></div>
|
|
<div><dt>Upstreams</dt><dd>5, on-demand</dd></div>
|
|
<div><dt>Platform</dt><dd>Talos / containerd</dd></div>
|
|
<div><dt>Open tests</dt><dd>6</dd></div>
|
|
</dl>
|
|
<p class="scrollcue" aria-hidden="true">scroll<span></span></p>
|
|
</header>
|
|
|
|
<div class="facts">
|
|
<dl>
|
|
<dt>Registry</dt><dd>zot v2.1.17 (digest-pinned image; latest upstream release at last check: v2.1.20)</dd>
|
|
<dt>Platform</dt><dd>Talos Linux v1.13.4, containerd 2.2.4, single-node Kubernetes</dd>
|
|
<dt>Upstreams</dt><dd>5 (Docker Hub, GHCR, Quay, registry.k8s.io, NGC), all on-demand pull-through</dd>
|
|
<dt>Storage</dt><dd>50Gi local PVC; dedupe on, GC on (1h delay / 24h interval); no retention config</dd>
|
|
<dt>Auth state</dt><dd>anonymous read enabled; per-node pull credentials staged, not yet enforced</dd>
|
|
<dt>Fallback</dt><dd>origin fallback ON (platform default; deliberately not disabled)</dd>
|
|
<dt>Reviewed</dt><dd>2026-08-25: live cluster configuration read; upstream docs and v2.1.17 source checked; live runtime tests listed separately in the open register</dd>
|
|
<dt>Open items</dt><dd>pull-aware digest-entry retention (needs v2.1.19+; blanket alternatives are capacity-heavy), auth canary negative test, upstream namespace prefixes, disconnected-serve drill</dd>
|
|
</dl>
|
|
<p style="margin:.9rem 0 0;font-size:.8rem;color:var(--ink-dim);font-family:var(--mono)">
|
|
Evidence labels used below: <span class="st ok">lab</span> reproduced here with a dated receipt ·
|
|
<span class="st ok">config</span> present in the inspected config ·
|
|
<span class="st ok">source</span> established from the deployed version's source ·
|
|
<span class="st ok">docs</span> stated in version-appropriate documentation ·
|
|
<span class="st open">reported</span> described in an unresolved upstream issue ·
|
|
<span class="st prop">proposed</span> not deployed · <span class="st open">open</span> not yet run</p>
|
|
</div>
|
|
|
|
<nav class="toc" aria-label="Section index">
|
|
<p class="kt">Index</p>
|
|
<ol>
|
|
<li><a href="#incident"><b>Section 01</b><span>The build that "hung"</span></a></li>
|
|
<li><a href="#anatomy"><b>Section 02</b><span>What the mirror does</span></a></li>
|
|
<li><a href="#routing"><b>Section 03</b><span>Routing and five upstreams</span></a></li>
|
|
<li><a href="#bootstrap"><b>Section 04</b><span>Bootstrap, fallback, failure</span></a></li>
|
|
<li><a href="#digests"><b>Section 05</b><span>Digests and supply chain</span></a></li>
|
|
<li><a href="#retention"><b>Section 06</b><span>Retention and the digest trap</span></a></li>
|
|
<li><a href="#auth"><b>Section 07</b><span>The authentication migration</span></a></li>
|
|
<li><a href="#observability"><b>Section 08</b><span>Observability that means something</span></a></li>
|
|
<li><a href="#production"><b>Section 09</b><span>The production translation</span></a></li>
|
|
<li><a href="#verified"><b>Section 10</b><span>Verified, open, references</span></a></li>
|
|
</ol>
|
|
</nav>
|
|
|
|
<h2 id="incident"><span class="n" aria-hidden="true">01</span>The build that "hung"</h2>
|
|
|
|
<div class="scene" id="s20"><div class="inner">
|
|
<img src="__M20__" alt="An armoured way-station machine exploded into parts: hexagonal hull, roof plate, a shelf of glowing bricks, intake pods and an output nozzle - the mirror as a physical machine.">
|
|
<div class="cap">The mirror as a machine: upstream intakes, one cache shelf, one serving nozzle.</div>
|
|
</div></div>
|
|
|
|
<p>I added zot after a CI pull hit Docker Hub's anonymous rate limit and the retry wrapper
|
|
hid the 429. At the time of the incident, this site's build-stage pull was anonymous.</p>
|
|
|
|
<p>Docker Hub does not go silent when it limits you. It answers with HTTP 429 and the error
|
|
code <code>toomanyrequests</code>. Under the
|
|
<a href="https://docs.docker.com/docker-hub/usage/pulls/">current published limits</a>,
|
|
unauthenticated users get 100 pulls per six hours per IPv4 address (or IPv6 /64); a
|
|
single-platform image counts as one pull, a multi-architecture image counts once per
|
|
architecture pulled, version checks do not count, and a <code>HEAD</code> request can read
|
|
the rate-limit headers without consuming a pull. A separate abuse limiter covers all
|
|
request types with its own 429 form. The <em>silence</em> in this incident was
|
|
manufactured on my side, by retries swallowing the answer.</p>
|
|
|
|
<p>The swallower, recovered from the build script's git history, was a double retry stack:
|
|
an outer shell wrapper allowing four attempts, wrapped around
|
|
<code>buildah build --retry 3</code>. Buildah's flag counts <em>retries</em> - one initial
|
|
try plus up to three more, applying to registry push/pull operations. Four outer attempts,
|
|
each containing up to four inner attempts: <strong>up to sixteen requests</strong> for one
|
|
repeatedly failing registry operation (a build with several pulls can produce more), each
|
|
billed against the limit it was trying to outlast. Retry amplification turns a throttle
|
|
into an outage and hides the evidence while doing it.</p>
|
|
|
|
<div class="pane-k">Receipt · the retry stack, as committed to the repo</div>
|
|
<pre class="pane">retry() {
|
|
local n=0 max=4
|
|
until "$@"; do
|
|
n=$((n+1)); [ "$n" -ge "$max" ] && { echo ">> failed after ${max} attempts" >&2; return 1; }
|
|
echo ">> attempt ${n} failed, retrying in $((n*8))s..." >&2; sleep $((n*8))
|
|
done
|
|
}
|
|
retry buildah build --retry 3 --retry-delay 5s ...
|
|
<span class="cm"># --retry 3 = one attempt + three retries; the wrapper multiplies that by four.</span>
|
|
<span class="cm"># A 429 answered every one of them, and the backoffs read as a hang.</span></pre>
|
|
|
|
<p>Pinning the base image by digest stopped the tag re-resolution but not the round-trip:
|
|
an ephemeral runner with a cold store still fetches the manifest behind that digest every
|
|
build. The repeated origin pulls justified a shared cache. This page takes it apart,
|
|
including the parts that turned out not to work the way I first believed.</p>
|
|
|
|
<div class="note"><span class="k">FIELD NOTE</span><br>The mirror did not fix the red X in
|
|
CI. That red was a separate upstream bug in the CI system's log-finalise step - cosmetic,
|
|
tolerated, documented. Two problems, one symptom; diagnose them separately.</div>
|
|
|
|
<h2 id="anatomy"><span class="n" aria-hidden="true">02</span>What the mirror actually does</h2>
|
|
|
|
<p>The mirror is zot: one registry, one 50Gi cache volume, digest-pinned like everything
|
|
it serves. Five public registries front it in <strong>on-demand pull-through</strong> mode -
|
|
a miss fetches and caches, a hit serves from the shelf. zot also offers polled mirroring
|
|
and pre-seeding; Docker Hub is on-demand only, per its documentation.</p>
|
|
|
|
<p>The storage block, from the live config:</p>
|
|
|
|
<pre>"storage": {
|
|
"rootDirectory": "/var/lib/registry",
|
|
"commit": true,
|
|
"dedupe": true,
|
|
"gc": true,
|
|
"gcDelay": "1h",
|
|
"gcInterval": "24h"
|
|
}</pre>
|
|
|
|
<p><code>commit</code> asks zot to commit writes to disk immediately instead of relying on
|
|
buffered flushing; it narrows the buffered-write window, while end-to-end power-loss
|
|
durability still depends on the filesystem, volume and disk. <code>dedupe</code> uses hard
|
|
links on local filesystem storage (remote backends implement it differently), saving
|
|
capacity when five upstreams ship the same base layers under different names - at the cost
|
|
of a startup reconciliation pass that matters operationally when toggled on existing data.
|
|
And <code>gc</code> with its two timers looks innocuous here; section 06 is about why it is
|
|
the most consequential block on this page.</p>
|
|
|
|
<div class="diagram" role="region" tabindex="0" aria-label="System context diagram"><span class="k">Diagram A · system context</span>
|
|
<svg viewBox="0 0 860 400" role="img" aria-labelledby="da-t da-d">
|
|
<title id="da-t">System context: consumers, the mirror, its storage, five upstreams, and the trust boundaries between them</title>
|
|
<desc id="da-d">CI and Kubernetes nodes pull from the zot mirror over the LAN. The mirror stores content on a 50Gi volume and syncs on demand from five upstream registries across the internet boundary. Anonymous read is currently allowed inbound; outbound upstream credentials live in one mounted secret. The origin-fallback path from nodes directly to upstreams is shown dashed.</desc>
|
|
<defs><marker id="da-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
|
|
<marker id="da-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker></defs>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
|
|
<rect x="12" y="60" width="150" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="87" y="82" fill="#dfe6f0">CI (buildah)</text><text x="87" y="99" fill="#98a2b4">bastion host</text>
|
|
<rect x="12" y="150" width="150" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="87" y="172" fill="#dfe6f0">k8s nodes</text><text x="87" y="189" fill="#98a2b4">containerd + local store</text>
|
|
<rect x="300" y="95" width="180" height="72" rx="8" fill="none" stroke="#dfe6f0"/><text x="390" y="122" fill="#dfe6f0">zot v2.1.17</text><text x="390" y="139" fill="#98a2b4">anon read (today)</text><text x="390" y="156" fill="#98a2b4">sync: onDemand x5</text>
|
|
<rect x="322" y="205" width="136" height="46" rx="8" fill="none" stroke="#98a2b4"/><text x="390" y="224" fill="#dfe6f0">50Gi PVC</text><text x="390" y="241" fill="#98a2b4">dedupe + GC 1h/24h</text>
|
|
<line x1="390" y1="167" x2="390" y2="200" stroke="#98a2b4"/>
|
|
<rect x="660" y="14" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="39" fill="#dfe6f0">docker.io</text>
|
|
<rect x="660" y="82" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="107" fill="#dfe6f0">ghcr.io</text>
|
|
<rect x="660" y="150" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="175" fill="#dfe6f0">quay.io</text>
|
|
<rect x="660" y="218" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="243" fill="#dfe6f0">registry.k8s.io</text>
|
|
<rect x="660" y="286" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="311" fill="#dfe6f0">nvcr.io</text>
|
|
<line x1="162" y1="86" x2="295" y2="115" stroke="#3fbaf5" marker-end="url(#da-a)"/>
|
|
<line x1="162" y1="176" x2="295" y2="145" stroke="#3fbaf5" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="110" x2="655" y2="40" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="120" x2="655" y2="100" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="131" x2="655" y2="168" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="142" x2="655" y2="235" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="153" x2="655" y2="300" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<path d="M162 200 Q 390 330 660 315" fill="none" stroke="#e879f9" stroke-dasharray="2 6" marker-end="url(#da-m)"/>
|
|
<text x="360" y="322" fill="#e879f9">origin fallback (dashed = only when the mirror cannot serve)</text>
|
|
<line x1="580" y1="8" x2="580" y2="345" stroke="#1c2434"/>
|
|
<text x="580" y="368" fill="#98a2b4">internet boundary - outbound creds in ONE mounted secret</text>
|
|
<text x="230" y="45" fill="#98a2b4">LAN - anonymous read today; per-node creds staged</text>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">solid = normal pull path · dashed cyan = on-demand sync on miss · dashed magenta = origin fallback</p>
|
|
<p class="why">One cache, two consumer classes; the two credential domains never mix.</p></div>
|
|
|
|
<p>Consumers point at it via the platform's machine-level registry config:</p>
|
|
|
|
<pre>machine:
|
|
registries:
|
|
mirrors:
|
|
docker.io:
|
|
endpoints:
|
|
- https://zot.bztmon.org
|
|
ghcr.io:
|
|
endpoints:
|
|
- https://zot.bztmon.org</pre>
|
|
|
|
<p>The mirror is the only listed endpoint, yet not a hard dependency: Talos tries
|
|
endpoints in order "and by default the last implicit endpoint is the original upstream
|
|
registry", unless <code>skipFallback: true</code>. An early version listed the upstream
|
|
explicitly as endpoint two; it was removed as redundant - origin fallback here is a Talos
|
|
default, not an endpoint this project maintains.</p>
|
|
|
|
<p>One asymmetry decides section 07's rollout order. Talos's
|
|
<a href="https://docs.siderolabs.com/talos/v1.13/reference/configuration/v1alpha1/config/">v1alpha1
|
|
machine-config reference</a> states for registry auth: "changes to the registry auth will
|
|
not be picked up by the CRI containerd plugin without a reboot" - matching what this lab
|
|
saw on v1.13.4. Mirror <em>endpoint</em> changes applied live here (same versions; the
|
|
docs are silent on that half, so it stays a lab observation, not a cross-version
|
|
guarantee). Secrets the registry consumes from a mount rotate with a pod restart. Put each
|
|
credential on the side that can move.</p>
|
|
|
|
<p>One pod, one PVC, one node is an accepted failure domain here, recorded as such;
|
|
section 09 covers what changes when the requirements do.</p>
|
|
|
|
<h2 id="routing"><span class="n" aria-hidden="true">03</span>Request routing and five upstreams</h2>
|
|
|
|
<p>Five registries feed one endpoint: when a node asks the mirror for
|
|
<code>pause:3.10</code>, how does it know which origin that means? Three facts:</p>
|
|
|
|
<ol>
|
|
<li><strong>containerd names the origin</strong> - mirror requests carry it as a query
|
|
parameter: <code>/v2/pause/manifests/3.10?ns=registry.k8s.io</code> (documented).</li>
|
|
<li><strong>zot v2.1.17 ignores it</strong> - no <code>ns</code> handling exists in the
|
|
deployed request path; using it for upstream selection is open feature request zot 4187.
|
|
The URL path alone selects the local repository.</li>
|
|
<li><strong>This deployment has no per-upstream prefixes</strong> - the live sync config
|
|
gives all five upstreams <code>prefix: "**"</code> and no <code>destination</code>. On a
|
|
miss, zot tries upstreams in config order until one has the path; the docs' own
|
|
multi-registry example instead gives each a distinct <code>destination</code>.</li>
|
|
</ol>
|
|
|
|
<div class="tablewrap" role="region" tabindex="0" aria-label="Upstream mapping table"><table>
|
|
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Upstream map - observed configuration</caption>
|
|
<tr><th scope="col">Origin</th><th scope="col">Local namespace</th><th scope="col">Outbound auth</th><th scope="col">Sync mode</th><th scope="col">Digest preserved</th><th scope="col">On miss</th><th scope="col">Status</th></tr>
|
|
<tr><td>docker.io</td><td rowspan="5">flat (no prefix - collision-ambiguous by construction)</td><td>Hub login (rate-limit lift)</td><td>onDemand</td><td>yes</td><td rowspan="5">upstreams tried in config order; first that resolves the path wins</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>ghcr.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>quay.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>registry.k8s.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>nvcr.io</td><td>$oauthtoken + key</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
|
|
</table></div>
|
|
|
|
<p>The flat namespace has not bitten because the origins use largely disjoint paths -
|
|
Hub's official images live under <code>library/</code> (the implicit prefix behind bare
|
|
names like <code>alpine</code>), NGC under <code>nvidia/</code>. Largely disjoint is a
|
|
probability: an organisation existing on both GHCR and Quay would collide, and config order
|
|
would pick the winner. Accepted risk, recorded; not a design.</p>
|
|
|
|
<div class="note warn"><span class="k">PROPOSED - NOT YET DEPLOYED</span><br>The safer
|
|
shape, documented on both halves: a distinct <code>destination</code> prefix per upstream
|
|
in zot (<code>/docker</code>, <code>/ghcr</code>, ...), with each Talos mirror pointed at
|
|
the prefixed path via <code>overridePath: true</code> (suppressing the automatic
|
|
<code>/v2</code> append). The requested path then names the origin and collisions become
|
|
impossible. Migration cost: every cached repository changes local path, so the cache
|
|
re-warms.</div>
|
|
|
|
<div class="diagram" role="region" tabindex="0" aria-label="Request flow diagram"><span class="k">Diagram B · request flow</span>
|
|
<svg viewBox="0 0 860 330" role="img" aria-labelledby="db-t db-d">
|
|
<title id="db-t">Request flow from a node through containerd to the mirror, with hit, miss, revalidation and fallback branches</title>
|
|
<desc id="db-d">A pull begins at the node's local content store. On local miss, containerd asks the mirror. A cached digest request is served from mirror storage without upstream contact. A tag request is revalidated against the upstream even when cached, on the deployed version. A storage miss triggers on-demand sync from the matching upstream. If the mirror cannot serve, containerd falls back to the origin registry.</desc>
|
|
<defs><marker id="db-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
|
|
<marker id="db-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker>
|
|
<marker id="db-g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#7fe0a7"/></marker></defs>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
|
|
<rect x="10" y="30" width="170" height="46" rx="8" fill="none" stroke="#3fbaf5"/><text x="95" y="49" fill="#dfe6f0">node content store</text><text x="95" y="66" fill="#98a2b4">hit = no network at all</text>
|
|
<rect x="10" y="130" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="95" y="155" fill="#dfe6f0">containerd -> mirror</text>
|
|
<rect x="300" y="40" width="200" height="44" rx="8" fill="none" stroke="#7fe0a7"/><text x="400" y="58" fill="#dfe6f0">digest request, cached</text><text x="400" y="75" fill="#7fe0a7">served locally (verified)</text>
|
|
<rect x="300" y="120" width="200" height="44" rx="8" fill="none" stroke="#e8b44a"/><text x="400" y="138" fill="#dfe6f0">tag request, cached</text><text x="400" y="155" fill="#e8b44a">still revalidates upstream</text>
|
|
<rect x="300" y="200" width="200" height="44" rx="8" fill="none" stroke="#3fbaf5"/><text x="400" y="218" fill="#dfe6f0">storage miss</text><text x="400" y="235" fill="#98a2b4">on-demand sync + cache</text>
|
|
<rect x="640" y="120" width="200" height="44" rx="8" fill="none" stroke="#98a2b4"/><text x="740" y="138" fill="#dfe6f0">origin registry</text><text x="740" y="155" fill="#98a2b4">rate limits live here</text>
|
|
<line x1="95" y1="76" x2="95" y2="125" stroke="#3fbaf5" marker-end="url(#db-a)"/><text x="150" y="105" fill="#98a2b4">local miss</text>
|
|
<line x1="180" y1="140" x2="295" y2="66" stroke="#7fe0a7" marker-end="url(#db-g)"/>
|
|
<line x1="180" y1="150" x2="295" y2="142" stroke="#e8b44a" marker-end="url(#db-a)"/>
|
|
<line x1="180" y1="160" x2="295" y2="218" stroke="#3fbaf5" marker-end="url(#db-a)"/>
|
|
<line x1="500" y1="142" x2="635" y2="142" stroke="#e8b44a" stroke-dasharray="5 4" marker-end="url(#db-a)"/>
|
|
<line x1="500" y1="222" x2="640" y2="158" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#db-a)"/>
|
|
<path d="M95 170 Q 95 300 400 300 L 660 300 Q 740 300 740 168" fill="none" stroke="#e879f9" stroke-dasharray="2 6" marker-end="url(#db-m)"/>
|
|
<text x="400" y="290" fill="#e879f9">containerd fallback: mirror endpoints exhausted -> origin direct</text>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">green = fully local · amber = the surprising branch on v2.1.17 · magenta = fallback</p>
|
|
<p class="why">The amber branch: a warm cache does not take the network out of the story.</p></div>
|
|
|
|
<h2 id="bootstrap"><span class="n" aria-hidden="true">04</span>Bootstrap, fallback and failure modes</h2>
|
|
|
|
<div class="scene" id="s21"><div class="inner">
|
|
<img src="__M21__" alt="Six machine parts arranged in a ring around an empty centre; one of them is a quarter-scale replica of the large way-station machine, representing the mirror's own image passing through the mirror.">
|
|
<div class="s21-ring" id="ring"></div>
|
|
<div class="cap">The dependency ring: one of the orbiting parts is the mirror itself - its own image is served through the thing it is.</div>
|
|
</div></div>
|
|
|
|
<p>Four circular dependencies live in this design; each got a different answer.</p>
|
|
|
|
<h3 id="loop1">Loop 1 - the mirror's own image comes through the mirror</h3>
|
|
<p>zot runs as a container whose image lives on a registry zot mirrors. When the hosting
|
|
node boots, it asks the mirror - which is not running, because the node is booting. The
|
|
loop breaks on the documented fallback: mirror endpoints exhaust, containerd falls through
|
|
to the origin. Keeping <code>skipFallback</code> unset is the load-bearing choice.</p>
|
|
|
|
<div class="note"><span class="k">FIELD NOTE · LAB, 2026-06</span><br>Exercised by
|
|
accident: a config change shipped with <code>preserveDigest</code> enabled but without its
|
|
mandatory partner <code>http.compat</code> - a pairing the registry refuses to start
|
|
without. The mirror crashlooped; the fleet fell through to upstream and nothing
|
|
user-visible broke. Since then, every config change runs the registry's own
|
|
<code>verify</code> in a throwaway pod before merging.</div>
|
|
|
|
<h3 id="loop2">Loop 2 - the recovery tooling ignores the mirror</h3>
|
|
<p>The fleet's rescue tooling does not pull through the mirror: its image is cached on the
|
|
operations host, outside the cluster, so it remains available while zot or its cluster is
|
|
down.</p>
|
|
|
|
<h3 id="loop3">Loop 3 - authentication cannot flip everywhere at once</h3>
|
|
<p>Enforcing auth needs every node to carry credentials that only apply after a reboot, so
|
|
anonymous read must survive until the last node is proven - and the proof must be designed
|
|
not to lie. Section 07 covers it.</p>
|
|
|
|
<h3 id="loop4">Loop 4 - the mirror's host is also its customer</h3>
|
|
<p>The node hosting the mirror boots its own workloads through it, including the tunnel
|
|
serving this page - so restarting that node carries a blast radius beyond the mirror, and
|
|
its reboot runbook lists every public-facing dependant and the order they return.</p>
|
|
|
|
<div class="diagram" role="region" tabindex="0" aria-label="Bootstrap paths diagram"><span class="k">Diagram D · two bootstrap worlds</span>
|
|
<svg viewBox="0 0 860 300" role="img" aria-labelledby="dd-t dd-d">
|
|
<title id="dd-t">Connected fail-open bootstrap versus disconnected preseeded bootstrap, as separate paths</title>
|
|
<desc id="dd-d">Left: the connected lab path - node boots, mirror miss, implicit fallback to origin, mirror comes up afterwards. Right: the disconnected path - no origin exists; the registry image and release set must be preseeded onto the host or imported from disk before anything else can start.</desc>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
|
|
<text x="215" y="28" fill="#3fbaf5">CONNECTED LAB - FAIL-OPEN (observed)</text>
|
|
<rect x="60" y="50" width="310" height="38" rx="8" fill="none" stroke="#3fbaf5"/><text x="215" y="74" fill="#dfe6f0">node boots, asks mirror: down</text>
|
|
<rect x="60" y="112" width="310" height="38" rx="8" fill="none" stroke="#e879f9"/><text x="215" y="136" fill="#dfe6f0">implicit fallback -> origin serves boot images</text>
|
|
<rect x="60" y="174" width="310" height="38" rx="8" fill="none" stroke="#3fbaf5"/><text x="215" y="198" fill="#dfe6f0">mirror starts (its image came via fallback)</text>
|
|
<rect x="60" y="236" width="310" height="38" rx="8" fill="none" stroke="#7fe0a7"/><text x="215" y="260" fill="#dfe6f0">estate converges back onto the mirror</text>
|
|
<line x1="215" y1="88" x2="215" y2="108" stroke="#98a2b4"/><line x1="215" y1="150" x2="215" y2="170" stroke="#98a2b4"/><line x1="215" y1="212" x2="215" y2="232" stroke="#98a2b4"/>
|
|
<text x="645" y="28" fill="#e8b44a">DISCONNECTED SITE - PRESEEDED (design, not deployed here)</text>
|
|
<rect x="490" y="50" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="74" fill="#dfe6f0">no origin exists - fallback is not a plan</text>
|
|
<rect x="490" y="112" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="136" fill="#dfe6f0">registry image preseeded / imported from disk</text>
|
|
<rect x="490" y="174" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="198" fill="#dfe6f0">release set, trust roots, creds staged locally</text>
|
|
<rect x="490" y="236" width="310" height="38" rx="8" fill="none" stroke="#7fe0a7"/><text x="645" y="260" fill="#dfe6f0">site serves itself; syncs on a schedule</text>
|
|
<line x1="645" y1="88" x2="645" y2="108" stroke="#98a2b4"/><line x1="645" y1="150" x2="645" y2="170" stroke="#98a2b4"/><line x1="645" y1="212" x2="645" y2="232" stroke="#98a2b4"/>
|
|
<line x1="430" y1="40" x2="430" y2="285" stroke="#1c2434"/>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">left = this lab, observed · right = the airgap translation, a design requirement not a deployed claim</p>
|
|
<p class="why">Loop 1 assumes an upstream to fall through to; the airgap deletes that
|
|
assumption and forces Loop 2's discipline onto the mirror itself.</p></div>
|
|
|
|
<p><strong>Fallback is a policy decision.</strong> This lab keeps it on for bootstrap
|
|
survivability and availability through mirror outages; the cost is that pulls can bypass
|
|
the mirror unnoticed and land on origin rate limits. A regulated or disconnected estate
|
|
makes the opposite call - <code>skipFallback: true</code>, egress restriction, preseeded
|
|
release stock, a tested recovery path - because there the bypass is the failure mode.
|
|
Failure classes differ in symptom, path and consequence:</p>
|
|
|
|
<div class="tablewrap" role="region" tabindex="0" aria-label="Failure mode matrix"><table>
|
|
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Failure modes - symptom, path, fallback and consequence</caption>
|
|
<tr><th scope="col">Failure</th><th scope="col">User-visible symptom</th><th scope="col">Request path</th><th scope="col">Fallback?</th><th scope="col">Detection</th><th scope="col">Security consequence</th></tr>
|
|
<tr><td>Mirror pod down</td><td>public pulls may continue transparently; private images, origin limits, DNS or TLS issues can slow or fail them</td><td>node -> origin</td><td>after the mirror endpoint fails; elapsed time differs by failure class (refused vs DNS vs TLS vs blackhole)</td><td>mirror probes red; origin egress rises</td><td>policy/audit bypass while down</td></tr>
|
|
<tr><td>Mirror up, sync wedged</td><td>pulls hang up to sync timeout</td><td>node -> mirror (blocked)</td><td>only after timeout</td><td>manifest probe stalls while /v2/ still answers</td><td>availability, not integrity</td></tr>
|
|
<tr><td>Upstream 429</td><td>miss/tag pulls fail or crawl</td><td>mirror -> origin refused</td><td>fallback also reaches the origin and may hit a pull or abuse limit - the quota bucket depends on the node's identity and source IP, not necessarily zot's</td><td>zot logs; retry storms amplify</td><td>self-inflicted denial of service via retries</td></tr>
|
|
<tr><td>Upstream down, digest cached</td><td>none expected</td><td>mirror serves locally</td><td>not needed</td><td>-</td><td>none (source-verified short-circuit; disconnected drill still open)</td></tr>
|
|
<tr><td>Upstream down, tag cached</td><td>reported on v2.1.17: failure or waiting until timeout, with content possibly served after it</td><td>mirror revalidates the tag upstream first</td><td>origin also down</td><td>the test matrix in section 08</td><td>availability surprise for anyone assuming cached means offline-safe (upstream-reported; not yet reproduced here)</td></tr>
|
|
<tr><td>PVC full</td><td>new pulls fail, cached OK-ish</td><td>mirror 5xx on writes</td><td>yes for missing content</td><td>capacity metrics (proposed)</td><td>availability; GC pressure</td></tr>
|
|
<tr><td>Digest entry GC-evicted</td><td>silent re-fetch on next pull</td><td>mirror -> origin re-sync</td><td>n/a</td><td>upstream egress for "cached" content</td><td>rate-limit exposure returns (section 06)</td></tr>
|
|
<tr><td>Node auth wrong (post-flip)</td><td>ImagePullBackOff</td><td>mirror 401; fallback unreliable on 401</td><td>unreliable</td><td>the canary gate (section 07)</td><td>the outage the gate exists to prevent</td></tr>
|
|
</table></div>
|
|
|
|
<h2 id="digests"><span class="n" aria-hidden="true">05</span>Digests, manifests and the supply chain</h2>
|
|
|
|
<p>Half the mistakes on this page came from conflating these objects:</p>
|
|
|
|
<ul>
|
|
<li>A <strong>tag</strong> is a mutable pointer; zero or many tags can reference one
|
|
manifest.</li>
|
|
<li>A <strong>manifest digest</strong> identifies exact manifest bytes; a multi-platform
|
|
<strong>index</strong>, each platform manifest, and every layer and config blob carry their
|
|
own.</li>
|
|
<li><code>name:tag@digest</code> is client-side grammar, not wire protocol: the OCI
|
|
distribution spec takes a tag <em>or</em> a digest in the URL. Clients resolve the digest
|
|
and ignore the tag (Kubernetes documents this), so digest-pinned pulls are immune to tag
|
|
moves. This page once mislearned the combined form as a cache-miss bug; retesting on
|
|
v2.1.17 showed the historical zot rejection no longer reproduces.</li>
|
|
<li>Digest pinning provides <strong>content integrity</strong> - not provenance, not
|
|
authorisation, not availability, and not freedom from network fetches on a cold store.</li>
|
|
</ul>
|
|
|
|
<p>For a mirror, digests carry one sharp operational rule: zot converts Docker-schema
|
|
manifests to OCI by default, and conversion changes the digest - breaking every digest pin
|
|
and signature downstream. The pairing that prevents it (<code>preserveDigest: true</code>
|
|
per upstream, with <code>http.compat: ["docker2s2"]</code>) is mandatory in both
|
|
directions: the registry refuses to start with one and not the other (the Loop 1 field
|
|
note is this rule being learned). This deployment runs both on all five upstreams.</p>
|
|
|
|
<p>Digest preservation and artifact completeness are different properties. The docs tie
|
|
<code>preserveDigest</code>/<code>compat</code> to keeping mirrored manifest bytes and
|
|
media types - and therefore signature <em>validity</em> - aligned with upstream. Whether
|
|
signature, SBOM and attestation objects <em>arrive</em> is separate: OCI 1.1 referrers
|
|
ride the <code>subject</code> field and their own API, legacy Cosign signatures ride
|
|
tag-schema conventions, and both depend on origin support and this version's sync
|
|
behaviour. This deployment has not verified a complete referrer graph for any origin and
|
|
performs no cryptographic signature verification; disconnected verification would also
|
|
need locally held trust material. Both gaps sit in the open register. Production estates
|
|
promote releases by digest and decide per policy whether referrers travel too; this lab
|
|
runs only the digest half.</p>
|
|
|
|
<h2 id="retention"><span class="n" aria-hidden="true">06</span>Retention, GC and the digest-only trap</h2>
|
|
|
|
<div class="scene" id="s23"><div class="inner">
|
|
<img src="__M23__" alt="A stuck intake pod ringed in magenta with three glowing cargo bricks queued behind it - cached content waiting on a jammed process.">
|
|
<div class="s23-pulse" id="pulse"></div>
|
|
<div class="cap">Cached bricks queued behind a jammed intake: availability problems here are quiet, not loud.</div>
|
|
</div></div>
|
|
|
|
<p>This section corrects the largest error in this page's first edition, which said "no
|
|
retention config means keep everything". True for tags; false for everything else - and
|
|
"everything else" includes the content this mirror exists to hold. The semantics, from
|
|
upstream documentation and the deployed version's source:</p>
|
|
|
|
<ul>
|
|
<li>With no retention configuration, <strong>all tags are retained</strong> and
|
|
<strong>all untagged manifests are deleted</strong> by GC (unless referenced by an index
|
|
or artifact). Timing is two-stage: the manifest ages past <code>gcDelay</code> (1h here),
|
|
then deletion happens at the next sweep (<code>gcInterval</code>, 24h here) - the survival
|
|
window depends on where creation falls relative to that sweep, not a fixed one-hour fuse.</li>
|
|
<li><strong>Digest-only pull-through entries are stored as untagged manifests.</strong>
|
|
That is the upstream fix's own problem statement: pull an image through the mirror by
|
|
digest, and the cache entry has no tag to protect it.</li>
|
|
</ul>
|
|
|
|
<p>Combined with section 04's revalidation behaviour: <em>tag</em> pulls are cached but
|
|
contact upstream anyway; <em>digest</em> pulls serve locally but their entries are
|
|
GC-eligible between sweeps - a digest-pinned fleet caches exactly the entry class GC may
|
|
delete. On v2.1.17 this mirror is a rate-limit shield and a latency win, <strong>not yet a
|
|
disconnection shelf</strong>; the first edition implied otherwise and was wrong.</p>
|
|
|
|
<div class="diagram" role="region" tabindex="0" aria-label="Storage and retention diagram"><span class="k">Diagram C · what keeps an object alive</span>
|
|
<svg viewBox="0 0 860 360" role="img" aria-labelledby="dc-t dc-d">
|
|
<title id="dc-t">The OCI object graph and which references protect content from garbage collection</title>
|
|
<desc id="dc-d">A tag points to an index; the index references platform manifests; manifests reference layers and a config blob. Referrers attach to a manifest via a subject field. A separate digest-only manifest sits with no tag pointing at it; it is eligible for garbage collection after the delay on the deployed version. Retention rules that keep tags do not protect the untagged manifest.</desc>
|
|
<defs><marker id="dc-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
|
|
<marker id="dc-b" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#98a2b4"/></marker></defs>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
|
|
<rect x="30" y="30" width="150" height="40" rx="8" fill="none" stroke="#7fe0a7"/><text x="105" y="55" fill="#dfe6f0">tag: v1.2</text>
|
|
<rect x="240" y="30" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="325" y="49" fill="#dfe6f0">index (multi-arch)</text><text x="325" y="64" fill="#98a2b4">own digest</text>
|
|
<rect x="470" y="10" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="555" y="35" fill="#dfe6f0">manifest amd64</text>
|
|
<rect x="470" y="62" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="555" y="87" fill="#dfe6f0">manifest arm64</text>
|
|
<rect x="700" y="10" width="140" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="770" y="35" fill="#dfe6f0">layers + config</text>
|
|
<rect x="700" y="62" width="140" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="770" y="87" fill="#dfe6f0">layers + config</text>
|
|
<rect x="470" y="130" width="170" height="40" rx="8" fill="none" stroke="#e879f9" stroke-dasharray="4 3"/><text x="555" y="149" fill="#dfe6f0">referrer (sig/SBOM)</text><text x="555" y="164" fill="#e879f9">subject -> manifest</text>
|
|
<line x1="180" y1="50" x2="235" y2="50" stroke="#7fe0a7" marker-end="url(#dc-a)"/>
|
|
<line x1="410" y1="42" x2="465" y2="30" stroke="#3fbaf5" marker-end="url(#dc-a)"/>
|
|
<line x1="410" y1="58" x2="465" y2="80" stroke="#3fbaf5" marker-end="url(#dc-a)"/>
|
|
<line x1="640" y1="30" x2="695" y2="30" stroke="#98a2b4" marker-end="url(#dc-b)"/>
|
|
<line x1="640" y1="82" x2="695" y2="82" stroke="#98a2b4" marker-end="url(#dc-b)"/>
|
|
<line x1="555" y1="130" x2="555" y2="106" stroke="#e879f9" stroke-dasharray="4 3" marker-end="url(#dc-a)"/>
|
|
<text x="105" y="120" fill="#7fe0a7">a tag is a keep-alive</text>
|
|
<rect x="30" y="215" width="280" height="52" rx="8" fill="none" stroke="#e8b44a"/><text x="170" y="236" fill="#dfe6f0">digest-only cached manifest</text><text x="170" y="253" fill="#e8b44a">NO tag references it</text>
|
|
<rect x="380" y="215" width="220" height="52" rx="8" fill="none" stroke="#ff8fa3"/><text x="490" y="236" fill="#dfe6f0">GC after gcDelay (1h here)</text><text x="490" y="253" fill="#ff8fa3">deleteUntagged default: true</text>
|
|
<line x1="310" y1="241" x2="375" y2="241" stroke="#ff8fa3" marker-end="url(#dc-a)"/>
|
|
<text x="430" y="310" fill="#98a2b4">keepTags protects TAGS in its repo. On v2.1.17/18 the amber box has one blanket lever: deleteUntagged:false,</text>
|
|
<text x="430" y="330" fill="#98a2b4">which keeps ALL untagged manifests (capacity trade). Selective pull-aware keepUntagged ships in v2.1.19.</text>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">green = protected by a tag · amber/red = the eviction path · dashed magenta = referrers (separate lifecycle)</p>
|
|
<p class="why">GC evaluates reference reachability: an untagged digest-only manifest can be
|
|
deleted even while its blobs remain present, and node-local image stores can mask the
|
|
eviction for days.</p></div>
|
|
|
|
<p>Retention policy semantics - two nearby systems use opposite matching rules:</p>
|
|
|
|
<ul>
|
|
<li>Retention policies match per repository, <strong>first match wins</strong> - order the
|
|
list most-specific-first. A repository matching no policy keeps its tags.</li>
|
|
<li>Within a matched repository, one <code>keepTags</code> rule inverts the default:
|
|
non-matching tags in <em>that repository</em> become deletable. The inversion is scoped to
|
|
the repository, not global - the first edition of this page overstated it.</li>
|
|
<li>Untagged manifests are governed separately (<code>deleteUntagged</code>, default true),
|
|
and on the deployed version no retention rule can protect them. Pull-aware
|
|
<code>keepUntagged</code> exists upstream from v2.1.19.</li>
|
|
<li>Authorisation policies (section 07) use <strong>longest match</strong>, not first
|
|
match. Two adjacent config blocks, two opposite precedence rules. Label which one you are
|
|
reasoning about.</li>
|
|
</ul>
|
|
|
|
<p><strong>Mitigations, ranked:</strong> upgrade to v2.1.19+ for pull-aware
|
|
<code>keepUntagged</code> (the designed fix, schema to be validated against the actual
|
|
binary before rollout); until then, widen <code>gcDelay</code>/<code>retention.delay</code>
|
|
(the upstream maintainer's interim suggestion), or set <code>deleteUntagged: false</code>,
|
|
which protects every untagged manifest at the cost of unbounded cache growth. Disabling GC
|
|
entirely swaps eviction for storage exhaustion with no reclaim path.</p>
|
|
|
|
<div class="note warn"><span class="k">SAFE VALIDATION - PROPOSED, NOT YET RUN</span><br>
|
|
The eviction claim is documented upstream and consistent with this config; it has not been
|
|
reproduced here. The reproduction, in a <em>disposable</em> instance only: record version
|
|
and sanitised config; pull by digest; confirm the stored manifest is untagged; shorten GC
|
|
timers; observe the manifest across a sweep; then block upstream and re-pull from a clean
|
|
runtime store, recording serve vs re-fetch. Two upstream warnings: the retention
|
|
verification tool executes orphan-blob GC for real even in dry-run, and local storage
|
|
requires the registry stopped. Never point retention experiments at live storage.</div>
|
|
|
|
<h2 id="auth"><span class="n" aria-hidden="true">07</span>The authentication migration</h2>
|
|
|
|
<div class="scene" id="s22"><div class="inner">
|
|
<img src="__M22__" alt="Four parts left to right: a waiting keyed cartridge, a key wedge, a tall gate frame, and a cartridge beyond the gate with its keyway lit - per-node credentials proven at a checkpoint.">
|
|
<div class="s22-glow" style="left:16%;top:30%" data-band="0"></div>
|
|
<div class="s22-glow" style="left:44%;top:28%" data-band="1"></div>
|
|
<div class="s22-glow" style="left:70%;top:26%" data-band="2"></div>
|
|
<div class="cap">The proving gate: a node counts as migrated when an authenticated pull succeeds where an anonymous one cannot.</div>
|
|
</div></div>
|
|
|
|
<p>Target state: the mirror refuses anonymous pulls. Current state: anonymous read on,
|
|
per-node credentials staged inert in machine configs until each reboot. The sequencing
|
|
problem has one central hazard: <strong>while anonymous read is on, an ordinary pull says
|
|
nothing about auth</strong> - a node whose credentials never applied is served as an
|
|
anonymous reader and false-passes the gate. The canary must live in a repository that
|
|
denies anonymous read, so success can only mean an authenticated pull.</p>
|
|
|
|
<p>The design is upstream-supported: authorisation resolves per-repository policies by
|
|
<strong>longest match</strong> (<code>**</code> is the default for anything unmatched), and
|
|
maintainer guidance confirms anonymous and authenticated access evaluate independently. So
|
|
a <code>canary/**</code> entry granting named identities read, with no
|
|
<code>anonymousPolicy</code>, denies anonymous on that path while the glob keeps the rest
|
|
of the shelf open. The staged policy:</p>
|
|
|
|
<pre>"accessControl": {
|
|
"repositories": {
|
|
"**": {
|
|
"anonymousPolicy": ["read"],
|
|
"policies": [
|
|
{ "users": ["zot-push"], "actions": ["read", "create", "update"] },
|
|
{ "users": ["zot-pull"], "actions": ["read"] }
|
|
]
|
|
},
|
|
"canary/**": {
|
|
"policies": [
|
|
{ "users": ["zot-pull", "zot-push"], "actions": ["read"] }
|
|
]
|
|
}
|
|
}
|
|
}</pre>
|
|
|
|
<div class="note"><span class="k">CLIENT CAVEAT · DOCUMENTED</span><br>Mixed
|
|
anonymous/authenticated policies trigger a Docker-client-specific workaround present in
|
|
this exact version: <code>/v2/</code> returns 401 to Docker user agents so the Docker CLI
|
|
sends credentials, meaning anonymous <em>docker</em> users must log in even for anonymous
|
|
repos. Podman and containerd are unaffected. This estate pulls with containerd, podman and
|
|
buildah, so the caveat is noted rather than felt.</div>
|
|
|
|
<p>The gate, per node, and what each step proves:</p>
|
|
|
|
<ol>
|
|
<li>Stage credentials in the node's machine config (inert; documented as requiring reboot).</li>
|
|
<li>Reboot the node at a planned window.</li>
|
|
<li><strong>Identity check:</strong> pull the protected canary through the node's own
|
|
runtime. Success = authenticated (anonymous cannot); 401 = back to step 1. The expected
|
|
split - 401 for missing/wrong credentials, 403 for a valid identity lacking the action -
|
|
sits in the open register, since the docs state the 403 case only for OIDC identities.</li>
|
|
<li><strong>Interception check, separately:</strong> the canary is a direct mirror
|
|
reference, so it says nothing about whether <code>docker.io/...</code> references route
|
|
through the mirror. That needs an original-upstream reference pulled on a node with a cold
|
|
runtime store, correlated with the mirror's request log - node-local content would satisfy
|
|
the pull without any network and fake a pass.</li>
|
|
<li>Anonymous read comes off the glob only when every node passes both checks. Rollback is
|
|
the previous config commit, named before the flip.</li>
|
|
</ol>
|
|
|
|
<div class="diagram" role="region" tabindex="0" aria-label="Authentication migration diagram"><span class="k">Diagram E · the migration ladder</span>
|
|
<svg viewBox="0 0 860 210" role="img" aria-labelledby="de-t de-d">
|
|
<title id="de-t">The authentication migration ladder with its per-node proof gate and rollback point</title>
|
|
<desc id="de-d">Five stages: anonymous baseline, credentials staged inert, node reboot, the two-part proof - authenticated canary pull plus logged mirror interception - and the final anonymous-off flip with a named rollback commit. A failing node loops from the proof back to staging.</desc>
|
|
<defs><marker id="de-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
|
|
<marker id="de-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker></defs>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="11" text-anchor="middle">
|
|
<rect x="10" y="70" width="140" height="48" rx="8" fill="none" stroke="#98a2b4"/><text x="80" y="90" fill="#dfe6f0">anon read ON</text><text x="80" y="106" fill="#98a2b4">baseline (today)</text>
|
|
<rect x="180" y="70" width="140" height="48" rx="8" fill="none" stroke="#3fbaf5"/><text x="250" y="90" fill="#dfe6f0">creds staged</text><text x="250" y="106" fill="#98a2b4">inert in config</text>
|
|
<rect x="350" y="70" width="140" height="48" rx="8" fill="none" stroke="#3fbaf5"/><text x="420" y="90" fill="#dfe6f0">node reboots</text><text x="420" y="106" fill="#98a2b4">auth goes live</text>
|
|
<rect x="520" y="70" width="160" height="48" rx="8" fill="none" stroke="#e879f9"/><text x="600" y="86" fill="#dfe6f0">two-part proof</text><text x="600" y="100" fill="#e879f9">canary + logged</text><text x="600" y="113" fill="#e879f9">interception</text>
|
|
<rect x="710" y="70" width="140" height="48" rx="8" fill="none" stroke="#7fe0a7"/><text x="780" y="90" fill="#dfe6f0">anon OFF</text><text x="780" y="106" fill="#98a2b4">rollback = prior commit</text>
|
|
<line x1="150" y1="94" x2="175" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
|
|
<line x1="320" y1="94" x2="345" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
|
|
<line x1="490" y1="94" x2="515" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
|
|
<line x1="680" y1="94" x2="705" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
|
|
<path d="M600 118 Q 600 175 480 175 L 320 175 Q 250 175 250 122" fill="none" stroke="#e879f9" stroke-dasharray="4 4" marker-end="url(#de-m)"/>
|
|
<text x="440" y="196" fill="#e879f9">401: node loops back - it does not count</text>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">the flip is per-fleet; the proof is per-node - one unproven node under a fleet-wide flip is an outage wearing a green tick</p>
|
|
<p class="why">Identity and interception fail independently; each has a false-pass mode
|
|
the other cannot detect.</p></div>
|
|
|
|
<p>Five trust domains never share material: node pull credentials (machine config,
|
|
reboot-bound), the push credential (used by the operator-run build host for registry
|
|
logins, not embedded in CI configuration), the mirror's upstream credentials (one mounted
|
|
secret, pod-restart-bound), TLS trust, and human admin access (SSO in front of the UI).
|
|
The push credential carries this page's oldest incident: in July 2026 its bcrypt hash and
|
|
plaintext drifted apart in the secrets manager, rotation became impossible, and pushes
|
|
stayed dead for twelve days - three applications shipped the day the rebuilt flow landed.
|
|
Today the hash and plaintext are separate entries in the same secrets manager, rotated as
|
|
a pair by a human; the registry consumes only the derived htpasswd file, each plaintext
|
|
reaches only its consumer, and the sync identity is read-only so automation cannot
|
|
half-rotate the pair. Residual cost, stated: one secrets-manager project holds both
|
|
halves, so its compromise yields verifier and credential together - versioned,
|
|
consumer-scoped secret objects remain a possible refinement.</p>
|
|
|
|
<h2 id="observability"><span class="n" aria-hidden="true">08</span>Observability that means something</h2>
|
|
|
|
<p>The worst incident in this mirror's life: an in-flight sync wedged, every pull from
|
|
that upstream hung, and the health endpoint returned 200 throughout:</p>
|
|
|
|
<div class="pane-k">Replay · observed 2026-07-07 · the stalled sync</div>
|
|
<pre class="pane">"image already demanded, waiting on channel"
|
|
$ kubectl -n zot rollout restart deploy zot <span class="cm"><- recovery action (not a root cause)</span>
|
|
<span class="cm"># probe an actual manifest, not the process's opinion of itself:</span>
|
|
$ curl -sfS --max-time 10 -o /dev/null -w '%{http_code}\n' \
|
|
https://zot.bztmon.org/v2/library/busybox/manifests/latest \
|
|
-H 'Accept: application/vnd.oci.image.index.v1+json'
|
|
<span class="good">200</span> <span class="cm"><- THIS is "the mirror works"</span></pre>
|
|
|
|
<p>The deployed source refines that story: the log line is coalescing by design -
|
|
concurrent requests join the first sync, which runs on a detached background context with
|
|
a three-hour default timeout and survives client disconnects. Waiters blocking on a
|
|
stalled sync until that timeout matches what we saw; the restart was recovery, and the
|
|
root cause was never isolated. Two knobs this config leaves unset - sync
|
|
<code>maxRetries</code> (disabled by default upstream) and a tighter
|
|
<code>syncTimeout</code> - are the first change on recurrence.</p>
|
|
|
|
<p>From the running version's source: <code>/livez</code>, <code>/readyz</code> and
|
|
<code>/startupz</code> exist as health endpoints (absent from this version's docs),
|
|
alongside the spec's <code>/v2/</code>. The Prometheus metrics extension exists upstream -
|
|
series names include <code>zot_http_requests_total</code>,
|
|
<code>zot_repo_storage_bytes</code>, <code>zot_repo_downloads_total</code> and
|
|
<code>zot_storage_lock_latency_seconds</code> - and is <strong>not enabled here</strong>;
|
|
wiring it, with scrape auth, is on the open list.</p>
|
|
|
|
<p>The layered checks, one question each:</p>
|
|
|
|
<div class="tablewrap" role="region" tabindex="0" aria-label="Layered operational checks"><table>
|
|
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Layered checks - one question each</caption>
|
|
<tr><th scope="col">Check</th><th scope="col">Question it answers</th><th scope="col">Status here</th></tr>
|
|
<tr><td><code>/livez</code> / <code>/readyz</code></td><td>is the process up / initialised</td><td><span class="st ok">available</span></td></tr>
|
|
<tr><td>authenticated manifest GET of a local-only canary</td><td>can the right identity read real content from local storage</td><td><span class="st prop">proposed</span></td></tr>
|
|
<tr><td>anonymous GET of the canary expecting 401</td><td>is the protection actually protecting</td><td><span class="st prop">proposed</span></td></tr>
|
|
<tr><td>original-reference pull on a cold node + mirror log line</td><td>is the runtime actually routed through the mirror</td><td><span class="st prop">proposed</span></td></tr>
|
|
<tr><td>digest pull with upstream blocked (disposable env)</td><td>does "cached" mean "served locally"</td><td><span class="st open">open - section 06</span></td></tr>
|
|
<tr><td>PVC usage + growth, GC activity, sync latency</td><td>when does capacity or eviction become the story</td><td><span class="st prop">needs metrics ext</span></td></tr>
|
|
<tr><td>certificate expiry, credential age</td><td>what breaks on a schedule</td><td><span class="st prop">proposed</span></td></tr>
|
|
</table></div>
|
|
|
|
<p>The caching test matrix (digest rows: source-verified and consistent with behaviour
|
|
here; disconnected rows: not yet run in this lab):</p>
|
|
|
|
<div class="tablewrap" role="region" tabindex="0" aria-label="Cache behaviour test matrix"><table>
|
|
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Cache behaviour - each row is an experiment</caption>
|
|
<tr><th scope="col">Scenario</th><th scope="col">Expected on v2.1.17</th><th scope="col">What proves it</th></tr>
|
|
<tr><td>cached tag, upstream reachable</td><td>served; upstream contacted anyway (revalidation)</td><td>zot log shows upstream request; latency includes round-trip</td></tr>
|
|
<tr><td>cached tag, upstream unreachable</td><td>fails or stalls until timeout, then local fallback path</td><td>pull timing vs sync timeout; error text</td></tr>
|
|
<tr><td>cached digest, upstream reachable</td><td>served locally, no upstream contact</td><td>absence of upstream request in zot logs during the pull</td></tr>
|
|
<tr><td>cached digest, upstream unreachable</td><td>served locally - IF the entry survived GC</td><td>the section-06 disposable-instance drill</td></tr>
|
|
<tr><td>cold node, warm mirror</td><td>mirror serves; node store fills</td><td>mirror log + no origin egress</td></tr>
|
|
<tr><td>warm node, evicted mirror entry</td><td>pull succeeds from node store - masking the eviction</td><td>this is the false-pass: only mirror-side inspection reveals it</td></tr>
|
|
<tr><td>multi-arch index + child manifests</td><td>index and per-platform manifests are separate cache entries</td><td>per-digest existence checks on the mirror</td></tr>
|
|
</table></div>
|
|
|
|
<h2 id="production"><span class="n" aria-hidden="true">09</span>The production translation</h2>
|
|
|
|
<div class="scene" id="s24"><div class="inner">
|
|
<img src="__M24__" alt="A monolithic vault with a wall of glowing bricks visible through its door, attended by four smaller self-sufficient way-stations - a central release source and site-local mirrors.">
|
|
<div class="cap">A central curated source and site-local registries: each site holds its own shelf, so a cut WAN idles nothing.</div>
|
|
</div></div>
|
|
|
|
<p>The first edition of this section asserted what "production" does; production chooses
|
|
from patterns against requirements:</p>
|
|
|
|
<ul>
|
|
<li><strong>Single registry, accepted failure domain</strong> - this lab's shape.
|
|
Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or a paused
|
|
deploy window (fallback off), with the cost accepted in writing.</li>
|
|
<li><strong>Sync-based HA</strong> - documented active/standby or active/active pairs of
|
|
<em>independent</em> instances, each with its own storage, mirroring each other behind a
|
|
load balancer. The caveat: the synchronisation window between polls bounds what failover
|
|
can lose.</li>
|
|
<li><strong>Scale-out clustering</strong> - instances shard repositories by hash and proxy
|
|
to the owner, classically over shared S3-compatible storage. Horizontal scale for load;
|
|
the shared storage becomes the availability story - a different property from the HA pair.</li>
|
|
<li><strong>Edge estates</strong> - a central source promotes releases by digest; each
|
|
site runs a local registry holding its release set on a schedule, so a severed WAN idles
|
|
nothing at pull time. A disconnection claim is valid only when content, required
|
|
referrers, trust roots, credentials and the registry's own bootstrap image are already
|
|
local (diagram D's right-hand column, per site).</li>
|
|
</ul>
|
|
|
|
<p>Whatever the topology: recovery time and point objectives get numbers before an
|
|
incident supplies them; backups are restore-tested copies off the failure domain (array
|
|
snapshots and RAID protect against disk loss, not against the array, the site or operator
|
|
error); cache capacity is planned from the section 06 retention policy; and the section 04
|
|
circular dependencies get drawn per site, because each exists at every scale - only the
|
|
cost of ignoring them changes.</p>
|
|
|
|
<h2 id="verified"><span class="n" aria-hidden="true">10</span>Verified, open, and where the claims come from</h2>
|
|
|
|
<div class="scene" id="s25"><div class="inner">
|
|
<img src="__M25__" alt="A wide flat node slab in an otherwise empty void; a single luminous brick hovers midway in its descent toward the slab - one small transfer of bytes as proof of life.">
|
|
<div class="cap">Proof of life is one authenticated byte transfer through the real path - not a status page.</div>
|
|
</div></div>
|
|
|
|
<p>When a node powers on, the fleet's admission check performs an authenticated manifest
|
|
pull through the node's own runtime before the node counts as a member: a process-health
|
|
response validates neither routing nor authorisation. Most of this page condenses into
|
|
that one command.</p>
|
|
|
|
<div class="tablewrap" role="region" tabindex="0" aria-label="Architecture decision table"><table>
|
|
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Architecture decisions</caption>
|
|
<tr><th scope="col">Decision</th><th scope="col">Chosen</th><th scope="col">Why</th><th scope="col">Trade-off</th><th scope="col">Rollback / alternative</th><th scope="col">Status</th></tr>
|
|
<tr><td>Origin fallback</td><td>ON (default kept)</td><td>bootstrap survivability, Loop 1</td><td>silent mirror bypass possible</td><td>skipFallback:true + preseed (airgap shape)</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>Sync mode</td><td>onDemand, all upstreams</td><td>cache follows real usage; Hub-safe</td><td>tag pulls revalidate upstream; digest entries untagged</td><td>polled sync for a curated release set</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>Digest preservation</td><td>preserveDigest + docker2s2, x5</td><td>digest pins + signatures survive mirroring</td><td>mandatory config pairing (crashloop scar)</td><td>none - non-negotiable for digest-pinned estates</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>Namespace layout</td><td>flat (no destinations)</td><td>simplicity at build time</td><td>ordered-trial upstream selection; theoretical collisions</td><td>per-origin destinations + overridePath</td><td><span class="st prop">redesign proposed</span></td></tr>
|
|
<tr><td>Retention</td><td>none configured</td><td>predates understanding the untagged rule</td><td>digest-only entries evict within hours</td><td>v2.1.19+ keepUntagged; interim wider gcDelay</td><td><span class="st open">correction owed</span></td></tr>
|
|
<tr><td>Auth</td><td>anonymous read until per-node proof</td><td>containerd 401-fallback unreliability makes big-bang flips dangerous</td><td>window with no pull auth</td><td>staged flip w/ canary gate + named rollback</td><td><span class="st open">in flight</span></td></tr>
|
|
<tr><td>Metrics</td><td>not enabled</td><td>minimal first deployment</td><td>capacity/eviction invisible</td><td>metrics extension + scrape auth</td><td><span class="st prop">proposed</span></td></tr>
|
|
</table></div>
|
|
|
|
<h3 id="open">Open verification register</h3>
|
|
<ol>
|
|
<li>Digest-entry eviction reproduction in a disposable instance (section 06 procedure).</li>
|
|
<li>Canary repo negative test: anonymous 401 on <code>canary/**</code> while anonymous
|
|
read elsewhere still succeeds; observed 401-vs-403 boundaries recorded.</li>
|
|
<li>Mirror interception proof: original-reference pull on a cold node correlated with the
|
|
mirror's request log.</li>
|
|
<li>Disconnected-serve drill: cached digest pull with upstream blocked, disposable
|
|
environment first.</li>
|
|
<li>Upgrade evaluation: v2.1.19/v2.1.20 for <code>keepUntagged</code>, with the retention
|
|
config written and reviewed before the upgrade, not after.</li>
|
|
<li>Namespace redesign migration plan (per-origin destinations + overridePath), including
|
|
cache re-warm cost.</li>
|
|
</ol>
|
|
|
|
<h3 id="refs">Verified against</h3>
|
|
<ul class="refs">
|
|
<li>zot documentation (versioned): <a href="https://zotregistry.dev/v2.1.18/articles/retention/">retention</a> · <a href="https://zotregistry.dev/v2.1.18/articles/storage/">storage</a> · <a href="https://zotregistry.dev/v2.1.18/articles/mirroring/">mirroring</a> · <a href="https://zotregistry.dev/v2.1.18/articles/authn-authz/">authn/authz</a> · <a href="https://zotregistry.dev/v2.1.18/articles/high-availability/">high availability</a> · <a href="https://zotregistry.dev/v2.1.18/articles/scaleout/">scale-out</a></li>
|
|
<li>zot upstream issues/PRs: <a href="https://github.com/project-zot/zot/issues/4148">4148 digest-only eviction</a> · <a href="https://github.com/project-zot/zot/pull/4191">4191 keepUntagged (v2.1.19)</a> · <a href="https://github.com/project-zot/zot/issues/4128">4128 tag revalidation</a> · <a href="https://github.com/project-zot/zot/issues/4187">4187 containerd ns param</a> · <a href="https://github.com/project-zot/zot/issues/2928">2928 mixed anonymous auth</a> · <a href="https://github.com/project-zot/zot/issues/2505">2505 demand coalescing</a></li>
|
|
<li>Talos: <a href="https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/images-container-runtime/pull-through-cache">pull-through registry cache (v1.13)</a> · containerd: <a href="https://github.com/containerd/containerd/blob/main/docs/hosts.md">registry hosts and the ns parameter</a></li>
|
|
<li>OCI: <a href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md">distribution spec</a> · <a href="https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/">image/distribution 1.1 (referrers)</a> · signing: <a href="https://docs.sigstore.dev/cosign/verifying/verify/">cosign verify</a></li>
|
|
<li>Docker Hub <a href="https://docs.docker.com/docker-hub/usage/">usage and limits</a> · buildah <a href="https://github.com/containers/buildah/blob/main/docs/buildah-build.1.md">build --retry semantics</a></li>
|
|
</ul>
|
|
|
|
|
|
|
|
<footer class="outro">
|
|
<p>© 2026 Jonathon Wright · <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC BY-NC-SA 4.0</a>
|
|
· hand-built static, shipped via GitOps · <a href="https://www.bztmon.com/">bztmon.com</a></p>
|
|
</footer>
|
|
|
|
</div>
|
|
|
|
<script src="__MIRROR_JS__"></script>
|