jwright
7f7035b98f
Content corrected against upstream docs, deployed-version source and the live config: retention/GC semantics (untagged manifests including digest-only pull-through entries are GC-eligible; keepUntagged is v2.1.19+, not deployed), warm-cache guarantees (tag pulls revalidate upstream per open issue; digest pulls serve locally), retry arithmetic (buildah --retry counts retries: 16 max, not 12), five-upstream flat-namespace routing stated as observed risk with a proposed prefix redesign, auth-canary design validated against longest-match policy precedence with the Docker-client caveat, coalescing reframed from deadlock to design with the restart as recovery-not-root-cause. Adds fact panel, contents, failure-mode and cache test matrices, decision table, open verification register and primary-source references. Front-end: grid full-bleed (no viewport units - no scrollbar overflow), shell-level skip link, real 1600x872 intrinsic dims injected from bytes, lazy below-fold plates, AA code-comment contrast, scene engine rewritten (IO-gated, rAF-coalesced, live reduced-motion, hidden-tab pause, zero guards). Assembler: per-page gates plus mirror-specific required/banned content assertions. README describes the multi-page site honestly.
877 lines
68 KiB
HTML
877 lines
68 KiB
HTML
<style>
|
|
:root{
|
|
--void:#070b14; --void-2:#0a0f1b; --ink:#dfe6f0; --ink-dim:#98a2b4;
|
|
--cyan:#3fbaf5; --magenta:#e879f9; --amber:#e8b44a; --line:#1c2434; --card:#0d1424;
|
|
--good:#7fe0a7; --bad:#ff8fa3;
|
|
--mono:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;
|
|
--sans:ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{scroll-behavior:smooth}
|
|
body{margin:0;background:var(--void);color:var(--ink);
|
|
font-family:var(--sans);font-size:1.05rem;line-height:1.65}
|
|
#bar{position:fixed;top:0;left:0;height:2px;width:0;background:linear-gradient(90deg,var(--cyan),var(--magenta));z-index:9}
|
|
/* full-bleed via named grid lines - no viewport units, so the vertical scrollbar
|
|
can never make the document wider than the visible viewport */
|
|
.layout{display:grid;
|
|
grid-template-columns:[full-start] minmax(1.25rem,1fr) [main-start] min(46rem,calc(100% - 2.5rem)) [main-end] minmax(1.25rem,1fr) [full-end]}
|
|
.layout>*{grid-column:main}
|
|
.layout>.scene{grid-column:full}
|
|
header.hero{min-height:70vh;display:flex;flex-direction:column;justify-content:center;padding:4.5rem 0 1.5rem;position:relative}
|
|
.eyebrow{font-family:var(--mono);font-size:.78rem;letter-spacing:.22em;color:var(--cyan);text-transform:uppercase}
|
|
h1{font-family:var(--sans);font-weight:750;font-size:clamp(2.8rem,8vw,4.6rem);
|
|
line-height:.98;margin:.5rem 0 1rem;text-wrap:balance;letter-spacing:-.01em}
|
|
.outcome{max-width:36rem;color:var(--ink-dim);font-size:1.12rem}
|
|
.outcome b{color:var(--ink);font-weight:600}
|
|
.backlink{font-family:var(--mono);font-size:.75rem;letter-spacing:.1em;margin-bottom:2rem}
|
|
.backlink a{color:var(--cyan);text-decoration:none}
|
|
.backlink a:hover{text-decoration:underline}
|
|
h2{font-family:var(--sans);font-weight:700;font-size:1.85rem;letter-spacing:0;
|
|
margin:4.2rem 0 1rem;text-wrap:balance;scroll-margin-top:2.5rem}
|
|
h2 .n{color:var(--cyan);font-family:var(--mono);font-weight:500;font-size:1rem;vertical-align:.35em;margin-right:.65rem;letter-spacing:.1em}
|
|
h3{font-weight:650;font-size:1.28rem;margin:2.2rem 0 .6rem;scroll-margin-top:2.5rem}
|
|
p{margin:.9rem 0}
|
|
code,.mono{font-family:var(--mono);font-size:.92em}
|
|
.scene{margin:3rem 0;overflow:hidden;position:relative}
|
|
.scene .inner{max-width:72rem;margin:0 auto;padding:0 1rem;position:relative}
|
|
.scene img{width:100%;height:auto;display:block;border-radius:.4rem}
|
|
.scene .cap{font-family:var(--mono);font-size:.74rem;color:var(--ink-dim);
|
|
letter-spacing:.1em;text-align:center;margin-top:.7rem;padding:0 1rem}
|
|
.sayit{border-left:3px solid var(--magenta);margin:2rem 0;padding:.4rem 0 .4rem 1.1rem;
|
|
font-size:1.26rem;font-weight:650;line-height:1.35;color:#fff}
|
|
.sayit .k{display:block;font-family:var(--mono);font-size:.7rem;letter-spacing:.2em;color:var(--magenta);margin-bottom:.3rem;font-weight:500}
|
|
.note{background:var(--card);border:1px solid var(--line);border-left:3px solid var(--cyan);
|
|
border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
|
|
.note .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan)}
|
|
.note.warn{border-left-color:var(--amber)}
|
|
.note.warn .k{color:var(--amber)}
|
|
.clause{background:linear-gradient(135deg,rgba(232,121,249,.07),transparent 60%);
|
|
border:1px solid var(--line);border-radius:.45rem;padding:1rem 1.2rem;margin:1.6rem 0;font-size:.98rem}
|
|
.clause .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--magenta)}
|
|
pre{background:#050810;border:1px solid var(--line);border-radius:.5rem;padding:1rem 1.1rem;
|
|
overflow-x:auto;font-family:var(--mono);font-size:.83rem;line-height:1.55;color:#c8d2e0}
|
|
pre.pane{border-left:3px solid var(--cyan)}
|
|
pre .cm{color:#8a95ab}
|
|
pre .good{color:var(--good)}
|
|
pre .bad{color:var(--bad)}
|
|
.pane-k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--ink-dim);
|
|
text-transform:uppercase;margin:1.6rem 0 .4rem}
|
|
.diagram{background:var(--card);border:1px solid var(--line);border-radius:.55rem;
|
|
padding:1.2rem;margin:2rem 0;overflow-x:auto}
|
|
.diagram .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan);display:block;margin-bottom:.8rem}
|
|
.diagram svg{width:100%;height:auto;display:block;min-width:560px}
|
|
.diagram .why{font-size:.9rem;color:var(--ink-dim);margin:.8rem 0 0}
|
|
.diagram .legend{font-family:var(--mono);font-size:.72rem;color:var(--ink-dim);margin:.6rem 0 0}
|
|
table{border-collapse:collapse;width:100%;font-size:.88rem;margin:1.4rem 0}
|
|
.tablewrap{overflow-x:auto;margin:1.4rem 0}
|
|
.tablewrap table{margin:0;min-width:640px}
|
|
th,td{border:1px solid var(--line);padding:.5rem .65rem;text-align:left;vertical-align:top}
|
|
th{font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;text-transform:uppercase;color:var(--cyan);background:var(--void-2)}
|
|
td .st{font-family:var(--mono);font-size:.7rem;letter-spacing:.06em;padding:.05rem .4rem;border-radius:.25rem;border:1px solid var(--line);white-space:nowrap}
|
|
.st.ok{color:var(--good);border-color:rgba(127,224,167,.4)}
|
|
.st.open{color:var(--amber);border-color:rgba(232,180,74,.4)}
|
|
.st.prop{color:var(--magenta);border-color:rgba(232,121,249,.4)}
|
|
.facts{border:1px solid var(--line);border-radius:.55rem;background:var(--card);
|
|
padding:1.1rem 1.3rem;margin:2.2rem 0;font-size:.92rem}
|
|
.facts dl{display:grid;grid-template-columns:max-content 1fr;gap:.35rem 1.2rem;margin:0}
|
|
.facts dt{font-family:var(--mono);font-size:.72rem;letter-spacing:.12em;text-transform:uppercase;color:var(--ink-dim);padding-top:.15rem}
|
|
.facts dd{margin:0}
|
|
.toc{border:1px solid var(--line);border-radius:.55rem;padding:1rem 1.3rem;margin:2rem 0;background:var(--void-2)}
|
|
.toc .k{font-family:var(--mono);font-size:.72rem;letter-spacing:.18em;color:var(--cyan);text-transform:uppercase}
|
|
.toc ol{margin:.6rem 0 0;padding-left:1.4rem;columns:2;column-gap:2.5rem;font-size:.92rem}
|
|
.toc li{margin:.3rem 0;break-inside:avoid}
|
|
.toc a{color:var(--ink);text-decoration:none}
|
|
.toc a:hover{color:var(--cyan)}
|
|
@media (max-width:640px){.toc ol{columns:1}}
|
|
ol{padding-left:1.3rem} ol li{margin:.6rem 0}
|
|
ul{padding-left:1.2rem} ul li{margin:.5rem 0}
|
|
em{color:#fff}
|
|
.traps p{margin:1.1rem 0}
|
|
.traps b,strong{color:#fff}
|
|
footer{margin:5rem 0 4rem;border-top:1px solid var(--line);padding-top:1.4rem;
|
|
color:var(--ink-dim);font-size:.88rem;font-family:var(--mono)}
|
|
a{color:var(--cyan)}
|
|
:focus-visible{outline:2px solid var(--cyan);outline-offset:2px}
|
|
.s21-ring{position:absolute;inset:0;pointer-events:none;mix-blend-mode:screen;opacity:.5;
|
|
background:conic-gradient(from 0deg at 50% 50%, transparent 0deg, rgba(63,186,245,.28) 24deg, transparent 60deg)}
|
|
.s22-glow{position:absolute;width:24%;aspect-ratio:1;border-radius:50%;pointer-events:none;
|
|
mix-blend-mode:screen;opacity:0;background:radial-gradient(circle,rgba(63,186,245,.5),transparent 65%)}
|
|
.s23-pulse{position:absolute;left:34%;top:44%;width:26%;aspect-ratio:1;border-radius:50%;pointer-events:none;
|
|
mix-blend-mode:screen;opacity:.25;background:radial-gradient(circle,rgba(232,121,249,.55),transparent 62%)}
|
|
.refs li{margin:.45rem 0;font-size:.92rem}
|
|
@media (prefers-reduced-motion: reduce){
|
|
html{scroll-behavior:auto}
|
|
.s21-ring,.s23-pulse{display:none}
|
|
.s22-glow{opacity:.35}
|
|
}
|
|
</style>
|
|
|
|
<div id="bar"></div>
|
|
|
|
<div class="layout">
|
|
|
|
<header class="hero">
|
|
<p class="backlink"><a href="/">← The Teaching Lab</a></p>
|
|
<p class="eyebrow">Dossier / 005 · operations</p>
|
|
<h1>The Mirror</h1>
|
|
<p class="outcome"><b>Outcome</b> — a pull-through OCI registry taken apart: why this
|
|
lab runs its own mirror, what the deployed version actually guarantees, the retention trap
|
|
that quietly evicts digest-pinned content, and how each lesson translates into production
|
|
platform engineering. Claims are labelled: observed here, documented upstream, or not yet
|
|
verified.</p>
|
|
</header>
|
|
|
|
<div class="facts">
|
|
<dl>
|
|
<dt>Registry</dt><dd>zot v2.1.17 (digest-pinned image; latest upstream release at last check: v2.1.20)</dd>
|
|
<dt>Platform</dt><dd>Talos Linux v1.13.4, containerd 2.2.4, single-node Kubernetes</dd>
|
|
<dt>Upstreams</dt><dd>5 (Docker Hub, GHCR, Quay, registry.k8s.io, NGC), all on-demand pull-through</dd>
|
|
<dt>Storage</dt><dd>50Gi local PVC; dedupe on, GC on (1h delay / 24h interval); no retention config</dd>
|
|
<dt>Auth state</dt><dd>anonymous read enabled; per-node pull credentials staged, not yet enforced</dd>
|
|
<dt>Fallback</dt><dd>origin fallback ON (platform default; deliberately not disabled)</dd>
|
|
<dt>Validated</dt><dd>2026-08-25 (live config read + upstream doc/source verification)</dd>
|
|
<dt>Open items</dt><dd>digest-entry retention (needs v2.1.19+), auth canary negative test, upstream namespace prefixes, disconnected-serve drill</dd>
|
|
</dl>
|
|
</div>
|
|
|
|
<nav class="toc" aria-labelledby="toc-h">
|
|
<p class="k" id="toc-h">Contents</p>
|
|
<ol>
|
|
<li><a href="#incident">The build that "hung"</a></li>
|
|
<li><a href="#anatomy">What the mirror actually does</a></li>
|
|
<li><a href="#routing">Request routing and five upstreams</a></li>
|
|
<li><a href="#bootstrap">Bootstrap, fallback and failure modes</a></li>
|
|
<li><a href="#digests">Digests, manifests and supply chain</a></li>
|
|
<li><a href="#retention">Retention, GC and the digest-only trap</a></li>
|
|
<li><a href="#auth">The authentication migration</a></li>
|
|
<li><a href="#observability">Observability that means something</a></li>
|
|
<li><a href="#production">The production translation</a></li>
|
|
<li><a href="#verified">Verified, open and references</a></li>
|
|
</ol>
|
|
</nav>
|
|
|
|
<h2 id="incident"><span class="n" aria-hidden="true">01</span>The build that "hung"</h2>
|
|
|
|
<div class="scene" id="s20"><div class="inner">
|
|
<img src="__M20__" alt="An armoured way-station machine exploded into parts: hexagonal hull, roof plate, a shelf of glowing bricks, intake pods and an output nozzle - the mirror as a physical machine.">
|
|
<div class="cap">The mirror as a machine: upstream intakes, one cache shelf, one serving nozzle.</div>
|
|
</div></div>
|
|
|
|
<p>Nobody sets out to run a registry. You inherit the need the day a CI build stalls for no
|
|
reason you can see.</p>
|
|
|
|
<p>This site's own build pulls one base image from Docker Hub, anonymously. One evening the
|
|
build sat there - no visible error, no progress. The accurate version of that story: Docker
|
|
Hub does not go silent when it limits you. It answers with an explicit HTTP 429 and the error
|
|
code <code>toomanyrequests</code>; under the current published limits, anonymous pulls get
|
|
100 manifest requests per six hours per IPv4 address, and the accounting is per manifest
|
|
<code>GET</code>. The <em>silence</em> was manufactured on my side, by retries swallowing
|
|
that answer.</p>
|
|
|
|
<p>The swallower, recovered from the build script's git history, was a double retry stack: an
|
|
outer shell wrapper allowing four attempts, wrapped around <code>buildah build --retry 3</code>.
|
|
Buildah's flag counts <em>retries</em>, not attempts - one initial try plus up to three
|
|
retries, and it applies to registry push/pull operations. Four outer attempts, each
|
|
containing up to four inner attempts: <strong>up to sixteen requests</strong> for one
|
|
consistently failing pull, each one billed against the same rate limit it was trying to
|
|
outlast. Retry amplification turns a throttle into an outage and hides the evidence.</p>
|
|
|
|
<div class="pane-k">Receipt · the retry stack, as committed to the repo</div>
|
|
<pre class="pane">retry() {
|
|
local n=0 max=4
|
|
until "$@"; do
|
|
n=$((n+1)); [ "$n" -ge "$max" ] && { echo ">> failed after ${max} attempts" >&2; return 1; }
|
|
echo ">> attempt ${n} failed, retrying in $((n*8))s..." >&2; sleep $((n*8))
|
|
done
|
|
}
|
|
retry buildah build --retry 3 --retry-delay 5s ...
|
|
<span class="cm"># --retry 3 = one attempt + three retries; the wrapper multiplies that by four.</span>
|
|
<span class="cm"># A 429 answered every one of them, and the backoffs read as a hang.</span></pre>
|
|
|
|
<p>The immediate fix was to pin the base image by digest - which stops tag re-resolution, but
|
|
does not remove the network round-trip: an ephemeral runner with a cold local store still
|
|
fetches the manifest behind that digest every build. The durable answer is structural:</p>
|
|
|
|
<div class="sayit"><span class="k">SAY IT</span>Why does a fleet of machines ask the public
|
|
internet for the same bytes, hundreds of times, forever?</div>
|
|
|
|
<p>A production platform answers with a mirror: one registry that fetches once and serves the
|
|
estate. The lab built one. This page takes it apart - including the parts that turned out
|
|
not to work the way I first believed.</p>
|
|
|
|
<div class="note"><span class="k">FIELD NOTE</span><br>The mirror did not fix the red X in
|
|
CI. That red was a separate upstream bug in the CI system's log-finalise step - cosmetic,
|
|
tolerated, documented. Two problems, one symptom. Diagnose them separately or you will fix
|
|
the wrong one and declare victory.</div>
|
|
|
|
<h2 id="anatomy"><span class="n" aria-hidden="true">02</span>What the mirror actually does</h2>
|
|
|
|
<p>The mirror is zot: a single OCI registry in its own namespace, one 50Gi cache volume,
|
|
running the same digest-pinned deployment discipline as everything it serves. It fronts five
|
|
public registries in <strong>on-demand pull-through</strong> mode: a miss fetches from
|
|
upstream and caches; a hit serves from the shelf. zot also supports polled mirroring
|
|
(periodic full sync of matching content) and pre-seeding; Docker Hub is on-demand only -
|
|
upstream documentation is explicit that polled mirroring should not be pointed at Hub.</p>
|
|
|
|
<p>The storage block, from the live config:</p>
|
|
|
|
<pre>"storage": {
|
|
"rootDirectory": "/var/lib/registry",
|
|
"commit": true,
|
|
"dedupe": true,
|
|
"gc": true,
|
|
"gcDelay": "1h",
|
|
"gcInterval": "24h"
|
|
}</pre>
|
|
|
|
<p><code>commit</code> fsyncs writes before acknowledging - crash safety. <code>dedupe</code>
|
|
hard-links identical blobs, which pays for itself when five upstreams ship the same base
|
|
layers under different names. And <code>gc</code> with its two timers looks innocuous here;
|
|
section 06 is about why it is the most consequential block on this page.</p>
|
|
|
|
<div class="diagram"><span class="k">Diagram A · system context</span>
|
|
<svg viewBox="0 0 860 400" role="img" aria-labelledby="da-t da-d">
|
|
<title id="da-t">System context: consumers, the mirror, its storage, five upstreams, and the trust boundaries between them</title>
|
|
<desc id="da-d">CI and Kubernetes nodes pull from the zot mirror over the LAN. The mirror stores content on a 50Gi volume and syncs on demand from five upstream registries across the internet boundary. Anonymous read is currently allowed inbound; outbound upstream credentials live in one mounted secret. The origin-fallback path from nodes directly to upstreams is shown dashed.</desc>
|
|
<defs><marker id="da-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
|
|
<marker id="da-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker></defs>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
|
|
<rect x="12" y="60" width="150" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="87" y="82" fill="#dfe6f0">CI (buildah)</text><text x="87" y="99" fill="#98a2b4">bastion host</text>
|
|
<rect x="12" y="150" width="150" height="52" rx="8" fill="none" stroke="#3fbaf5"/><text x="87" y="172" fill="#dfe6f0">k8s nodes</text><text x="87" y="189" fill="#98a2b4">containerd + local store</text>
|
|
<rect x="300" y="95" width="180" height="72" rx="8" fill="none" stroke="#dfe6f0"/><text x="390" y="122" fill="#dfe6f0">zot v2.1.17</text><text x="390" y="139" fill="#98a2b4">anon read (today)</text><text x="390" y="156" fill="#98a2b4">sync: onDemand x5</text>
|
|
<rect x="322" y="205" width="136" height="46" rx="8" fill="none" stroke="#98a2b4"/><text x="390" y="224" fill="#dfe6f0">50Gi PVC</text><text x="390" y="241" fill="#98a2b4">dedupe + GC 1h/24h</text>
|
|
<line x1="390" y1="167" x2="390" y2="200" stroke="#98a2b4"/>
|
|
<rect x="660" y="14" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="39" fill="#dfe6f0">docker.io</text>
|
|
<rect x="660" y="82" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="107" fill="#dfe6f0">ghcr.io</text>
|
|
<rect x="660" y="150" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="175" fill="#dfe6f0">quay.io</text>
|
|
<rect x="660" y="218" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="243" fill="#dfe6f0">registry.k8s.io</text>
|
|
<rect x="660" y="286" width="186" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="753" y="311" fill="#dfe6f0">nvcr.io</text>
|
|
<line x1="162" y1="86" x2="295" y2="115" stroke="#3fbaf5" marker-end="url(#da-a)"/>
|
|
<line x1="162" y1="176" x2="295" y2="145" stroke="#3fbaf5" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="110" x2="655" y2="40" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="120" x2="655" y2="100" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="131" x2="655" y2="168" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="142" x2="655" y2="235" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<line x1="480" y1="153" x2="655" y2="300" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#da-a)"/>
|
|
<path d="M162 200 Q 390 330 660 315" fill="none" stroke="#e879f9" stroke-dasharray="2 6" marker-end="url(#da-m)"/>
|
|
<text x="360" y="322" fill="#e879f9">origin fallback (dashed = only when the mirror cannot serve)</text>
|
|
<line x1="580" y1="8" x2="580" y2="345" stroke="#1c2434"/>
|
|
<text x="580" y="368" fill="#98a2b4">internet boundary - outbound creds in ONE mounted secret</text>
|
|
<text x="230" y="45" fill="#98a2b4">LAN - anonymous read today; per-node creds staged</text>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">solid = normal pull path · dashed cyan = on-demand sync on miss · dashed magenta = origin fallback</p>
|
|
<p class="why">The decision this explains: one cache serves two consumer classes, and the
|
|
two credential domains (nodes-to-mirror, mirror-to-upstreams) never mix.</p></div>
|
|
|
|
<p>Consumers reach the mirror through the platform's machine-level registry config:</p>
|
|
|
|
<pre>machine:
|
|
registries:
|
|
mirrors:
|
|
docker.io:
|
|
endpoints:
|
|
- https://zot.bztmon.org
|
|
ghcr.io:
|
|
endpoints:
|
|
- https://zot.bztmon.org</pre>
|
|
|
|
<p>The mirror is the only listed endpoint, and that is still not a hard dependency, because
|
|
Talos documents an implicit final fallback: endpoints are tried in order, "and by default
|
|
the last implicit endpoint is the original upstream registry", unless
|
|
<code>skipFallback: true</code> says otherwise. The lab's first version listed the upstream
|
|
as an explicit second endpoint; the refinement that removed it came from reading the
|
|
behaviour properly - the default already guaranteed it. Know which of your safety nets you
|
|
built, and which are defaults you merely have not broken.</p>
|
|
|
|
<p>One asymmetry decides the whole rollout order in section 07: registry <em>auth</em> in
|
|
the machine config is documented by Talos as requiring a reboot before the CRI picks it up.
|
|
Mirror <em>endpoint</em> changes applied live in this lab (observed on Talos v1.13.4 /
|
|
containerd 2.2.4; the docs are silent on this half, so treat it as an observation, not a
|
|
guarantee). Credentials the registry consumes from a mounted secret rotate with a pod
|
|
restart. Put each credential on the side that can move.</p>
|
|
|
|
<div class="clause"><span class="k">THE HOMELAB CLAUSE</span><br>One pod, one PVC, one node
|
|
is an accepted failure domain, not a pattern. Section 09 covers what changes when the
|
|
requirements change - and what genuinely does not.</div>
|
|
|
|
<h2 id="routing"><span class="n" aria-hidden="true">03</span>Request routing and five upstreams</h2>
|
|
|
|
<p>Five registries feed one endpoint, which raises a question the first version of this page
|
|
skated past: when a node asks the mirror for <code>pause:3.10</code>, how does the mirror
|
|
know whether that means Docker Hub, GHCR, Quay, registry.k8s.io or NGC?</p>
|
|
|
|
<p>Three facts, all verified:</p>
|
|
|
|
<ol>
|
|
<li><strong>containerd tells the mirror where the request came from</strong> - a mirror
|
|
request carries the original registry as a query parameter:
|
|
<code>/v2/pause/manifests/3.10?ns=registry.k8s.io</code>. That is documented containerd
|
|
behaviour.</li>
|
|
<li><strong>zot v2.1.17 ignores it.</strong> There is no handling of the <code>ns</code>
|
|
parameter in the deployed version's request path, and using it for upstream selection is an
|
|
open upstream feature request (zot issue 4187) with, at last check, no maintainer response.
|
|
The URL path alone selects the local repository.</li>
|
|
<li><strong>This deployment has no per-upstream prefixes.</strong> The live sync config
|
|
gives all five upstreams <code>prefix: "**"</code> and no <code>destination</code> - one
|
|
flat namespace. On a miss, zot tries the configured upstreams in order until one has the
|
|
path. Upstream documentation's own multi-registry example instead gives each upstream a
|
|
distinct <code>destination</code> so the requested path selects the origin.</li>
|
|
</ol>
|
|
|
|
<div class="tablewrap"><table>
|
|
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Upstream map - observed configuration</caption>
|
|
<tr><th>Origin</th><th>Local namespace</th><th>Outbound auth</th><th>Sync mode</th><th>Digest preserved</th><th>On miss</th><th>Status</th></tr>
|
|
<tr><td>docker.io</td><td rowspan="5">flat (no prefix - collision-ambiguous by construction)</td><td>Hub login (rate-limit lift)</td><td>onDemand</td><td>yes</td><td rowspan="5">upstreams tried in config order; first that resolves the path wins</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>ghcr.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>quay.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>registry.k8s.io</td><td>none</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>nvcr.io</td><td>$oauthtoken + key</td><td>onDemand</td><td>yes</td><td><span class="st ok">observed</span></td></tr>
|
|
</table></div>
|
|
|
|
<p>Why has the flat namespace not bitten? Because the five origins use largely disjoint path
|
|
conventions - Hub's official images live under <code>library/</code> (the implicit prefix
|
|
behind bare names like <code>alpine</code>), NGC content sits under <code>nvidia/</code>,
|
|
registry.k8s.io has its own layout. "Largely disjoint" is a probability, not a guarantee: an
|
|
organisation name that exists on both GHCR and Quay would collide silently, and the winner
|
|
would be config order. That is an accepted risk in a lab; it is not a design.</p>
|
|
|
|
<div class="note warn"><span class="k">PROPOSED - NOT YET DEPLOYED</span><br>The safer shape,
|
|
verified against the documentation of both halves but not yet applied here: give each
|
|
upstream a distinct <code>destination</code> prefix in zot
|
|
(<code>/docker</code>, <code>/ghcr</code>, ...), and point each Talos mirror at the
|
|
prefixed path with <code>overridePath: true</code> (which stops the automatic
|
|
<code>/v2</code> append so the prefix survives). Then the requested path itself names the
|
|
origin, and collisions become impossible rather than improbable. The migration cost: every
|
|
cached repository changes its local path, so the cache re-warms.</div>
|
|
|
|
<div class="diagram"><span class="k">Diagram B · request flow</span>
|
|
<svg viewBox="0 0 860 330" role="img" aria-labelledby="db-t db-d">
|
|
<title id="db-t">Request flow from a node through containerd to the mirror, with hit, miss, revalidation and fallback branches</title>
|
|
<desc id="db-d">A pull begins at the node's local content store. On local miss, containerd asks the mirror. A cached digest request is served from mirror storage without upstream contact. A tag request is revalidated against the upstream even when cached, on the deployed version. A storage miss triggers on-demand sync from the matching upstream. If the mirror cannot serve, containerd falls back to the origin registry.</desc>
|
|
<defs><marker id="db-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
|
|
<marker id="db-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker>
|
|
<marker id="db-g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#7fe0a7"/></marker></defs>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
|
|
<rect x="10" y="30" width="170" height="46" rx="8" fill="none" stroke="#3fbaf5"/><text x="95" y="49" fill="#dfe6f0">node content store</text><text x="95" y="66" fill="#98a2b4">hit = no network at all</text>
|
|
<rect x="10" y="130" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="95" y="155" fill="#dfe6f0">containerd -> mirror</text>
|
|
<rect x="300" y="40" width="200" height="44" rx="8" fill="none" stroke="#7fe0a7"/><text x="400" y="58" fill="#dfe6f0">digest request, cached</text><text x="400" y="75" fill="#7fe0a7">served locally (verified)</text>
|
|
<rect x="300" y="120" width="200" height="44" rx="8" fill="none" stroke="#e8b44a"/><text x="400" y="138" fill="#dfe6f0">tag request, cached</text><text x="400" y="155" fill="#e8b44a">still revalidates upstream</text>
|
|
<rect x="300" y="200" width="200" height="44" rx="8" fill="none" stroke="#3fbaf5"/><text x="400" y="218" fill="#dfe6f0">storage miss</text><text x="400" y="235" fill="#98a2b4">on-demand sync + cache</text>
|
|
<rect x="640" y="120" width="200" height="44" rx="8" fill="none" stroke="#98a2b4"/><text x="740" y="138" fill="#dfe6f0">origin registry</text><text x="740" y="155" fill="#98a2b4">rate limits live here</text>
|
|
<line x1="95" y1="76" x2="95" y2="125" stroke="#3fbaf5" marker-end="url(#db-a)"/><text x="150" y="105" fill="#98a2b4">local miss</text>
|
|
<line x1="180" y1="140" x2="295" y2="66" stroke="#7fe0a7" marker-end="url(#db-g)"/>
|
|
<line x1="180" y1="150" x2="295" y2="142" stroke="#e8b44a" marker-end="url(#db-a)"/>
|
|
<line x1="180" y1="160" x2="295" y2="218" stroke="#3fbaf5" marker-end="url(#db-a)"/>
|
|
<line x1="500" y1="142" x2="635" y2="142" stroke="#e8b44a" stroke-dasharray="5 4" marker-end="url(#db-a)"/>
|
|
<line x1="500" y1="222" x2="640" y2="158" stroke="#3fbaf5" stroke-dasharray="5 4" marker-end="url(#db-a)"/>
|
|
<path d="M95 170 Q 95 300 400 300 L 660 300 Q 740 300 740 168" fill="none" stroke="#e879f9" stroke-dasharray="2 6" marker-end="url(#db-m)"/>
|
|
<text x="400" y="290" fill="#e879f9">containerd fallback: mirror endpoints exhausted -> origin direct</text>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">green = fully local · amber = the surprising branch on v2.1.17 · magenta = fallback</p>
|
|
<p class="why">The amber branch is section 04's punchline: a warm cache does not mean the
|
|
network is out of the story.</p></div>
|
|
|
|
<h2 id="bootstrap"><span class="n" aria-hidden="true">04</span>Bootstrap, fallback and failure modes</h2>
|
|
|
|
<div class="scene" id="s21"><div class="inner">
|
|
<img src="__M21__" alt="Six machine parts arranged in a ring around an empty centre; one of them is a quarter-scale replica of the large way-station machine, representing the mirror's own image passing through the mirror.">
|
|
<div class="s21-ring" id="ring"></div>
|
|
<div class="cap">The dependency ring: one of the orbiting parts is the mirror itself - its own image is served through the thing it is.</div>
|
|
</div></div>
|
|
|
|
<p>Every infrastructure service eventually meets the question: what do you depend on, and
|
|
what happens when you ARE the dependency? The mirror has four such loops, and each got a
|
|
different answer.</p>
|
|
|
|
<h3 id="loop1">Loop 1 - the mirror's own image comes through the mirror</h3>
|
|
<p>zot runs as a container whose image lives on a registry zot mirrors. When the hosting
|
|
node boots, it asks the mirror - which is not running, because the node is booting. The
|
|
loop breaks on the documented fallback: mirror endpoints exhaust, containerd falls through
|
|
to the origin. The design decision is restraint - <em>not</em> setting
|
|
<code>skipFallback: true</code>.</p>
|
|
|
|
<div class="note"><span class="k">FIELD NOTE · OBSERVED</span><br>Proven by accident:
|
|
a config change shipped with <code>preserveDigest</code> enabled but without its mandatory
|
|
partner <code>http.compat</code> - a pairing the registry refuses to start without (the
|
|
documentation is explicit; so was the crashloop). The mirror went down; the fleet quietly
|
|
fell through to upstream and nothing user-visible broke. An unplanned failover exercise,
|
|
passed. The standing rule it bought: validate the config with the registry's own
|
|
<code>verify</code> in a throwaway pod before merging, every time.</div>
|
|
|
|
<h3 id="loop2">Loop 2 - the recovery tooling deliberately ignores the mirror</h3>
|
|
<p>The fleet's rescue tooling could pull its execution image through the mirror like
|
|
everything else. It does not: its image is cached on the operations host, outside the
|
|
cluster. A recovery tool that depends on the thing it recovers is not a recovery tool.</p>
|
|
|
|
<h3 id="loop3">Loop 3 - authentication cannot flip everywhere at once</h3>
|
|
<p>Covered properly in section 07; the loop shape: enforcing auth requires every node to
|
|
carry credentials that only apply after a reboot, so anonymous read must survive until the
|
|
last node is proven, and the proof itself must be designed not to lie.</p>
|
|
|
|
<h3 id="loop4">Loop 4 - the mirror's host is also the mirror's customer</h3>
|
|
<p>The node hosting the mirror boots its own workloads through it - including the tunnel
|
|
that serves this very page. "Restart the mirror's node" therefore carries a blast radius
|
|
far beyond the mirror, and the runbook for that reboot lists every public-facing thing
|
|
riding on it and the order they return. Draw the dependency arrows for your own estate; the
|
|
ones that surprise you are the ones that will page you.</p>
|
|
|
|
<div class="diagram"><span class="k">Diagram D · two bootstrap worlds</span>
|
|
<svg viewBox="0 0 860 300" role="img" aria-labelledby="dd-t dd-d">
|
|
<title id="dd-t">Connected fail-open bootstrap versus disconnected preseeded bootstrap, as separate paths</title>
|
|
<desc id="dd-d">Left: the connected lab path - node boots, mirror miss, implicit fallback to origin, mirror comes up afterwards. Right: the disconnected path - no origin exists; the registry image and release set must be preseeded onto the host or imported from disk before anything else can start.</desc>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
|
|
<text x="215" y="28" fill="#3fbaf5">CONNECTED LAB - FAIL-OPEN (observed)</text>
|
|
<rect x="60" y="50" width="310" height="38" rx="8" fill="none" stroke="#3fbaf5"/><text x="215" y="74" fill="#dfe6f0">node boots, asks mirror: down</text>
|
|
<rect x="60" y="112" width="310" height="38" rx="8" fill="none" stroke="#e879f9"/><text x="215" y="136" fill="#dfe6f0">implicit fallback -> origin serves boot images</text>
|
|
<rect x="60" y="174" width="310" height="38" rx="8" fill="none" stroke="#3fbaf5"/><text x="215" y="198" fill="#dfe6f0">mirror starts (its image came via fallback)</text>
|
|
<rect x="60" y="236" width="310" height="38" rx="8" fill="none" stroke="#7fe0a7"/><text x="215" y="260" fill="#dfe6f0">estate converges back onto the mirror</text>
|
|
<line x1="215" y1="88" x2="215" y2="108" stroke="#98a2b4"/><line x1="215" y1="150" x2="215" y2="170" stroke="#98a2b4"/><line x1="215" y1="212" x2="215" y2="232" stroke="#98a2b4"/>
|
|
<text x="645" y="28" fill="#e8b44a">DISCONNECTED SITE - PRESEEDED (design, not deployed here)</text>
|
|
<rect x="490" y="50" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="74" fill="#dfe6f0">no origin exists - fallback is not a plan</text>
|
|
<rect x="490" y="112" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="136" fill="#dfe6f0">registry image preseeded / imported from disk</text>
|
|
<rect x="490" y="174" width="310" height="38" rx="8" fill="none" stroke="#e8b44a"/><text x="645" y="198" fill="#dfe6f0">release set, trust roots, creds staged locally</text>
|
|
<rect x="490" y="236" width="310" height="38" rx="8" fill="none" stroke="#7fe0a7"/><text x="645" y="260" fill="#dfe6f0">site serves itself; syncs on a schedule</text>
|
|
<line x1="645" y1="88" x2="645" y2="108" stroke="#98a2b4"/><line x1="645" y1="150" x2="645" y2="170" stroke="#98a2b4"/><line x1="645" y1="212" x2="645" y2="232" stroke="#98a2b4"/>
|
|
<line x1="430" y1="40" x2="430" y2="285" stroke="#1c2434"/>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">left = this lab, observed · right = the airgap translation, a design requirement not a deployed claim</p>
|
|
<p class="why">Loop 1's trick assumes an upstream exists to fall through to. The airgap
|
|
deletes that assumption, and forces Loop 2's discipline onto the mirror itself.</p></div>
|
|
|
|
<p><strong>Fallback is a policy decision, not a universal win.</strong> This lab keeps it on:
|
|
better bootstrap survivability, availability through mirror outages, and the cost is real -
|
|
pulls can silently bypass the mirror (losing cache benefit and any future policy point) and
|
|
land on origin rate limits. A regulated or disconnected estate makes the opposite call:
|
|
<code>skipFallback: true</code>, egress restrictions, preseeded release stock, and a tested
|
|
recovery path, because silent bypass is the failure mode there, not the safety net.</p>
|
|
|
|
<p>And "fails fast" is not one behaviour. The failure modes differ in symptom, path and
|
|
consequence:</p>
|
|
|
|
<div class="tablewrap"><table>
|
|
<tr><th>Failure</th><th>User-visible symptom</th><th>Request path</th><th>Fallback?</th><th>Detection</th><th>Security consequence</th></tr>
|
|
<tr><td>Mirror pod down</td><td>none (pulls slower)</td><td>node -> origin</td><td>yes, immediate</td><td>mirror probes red; origin egress rises</td><td>policy/audit bypass while down</td></tr>
|
|
<tr><td>Mirror up, sync wedged</td><td>pulls hang up to sync timeout</td><td>node -> mirror (blocked)</td><td>only after timeout</td><td>manifest probe stalls while /v2/ still answers</td><td>availability, not integrity</td></tr>
|
|
<tr><td>Upstream 429</td><td>miss/tag pulls fail or crawl</td><td>mirror -> origin refused</td><td>fallback hits the same limit</td><td>zot logs; retry storms amplify</td><td>none; self-inflicted DoS via retries</td></tr>
|
|
<tr><td>Upstream down, digest cached</td><td>none</td><td>mirror serves locally</td><td>not needed</td><td>-</td><td>none (verified behaviour)</td></tr>
|
|
<tr><td>Upstream down, tag cached</td><td>fails/stalls on v2.1.17</td><td>mirror insists on revalidating</td><td>origin also down</td><td>the test matrix in section 08</td><td>availability surprise - the trap of assuming "cached = offline-safe"</td></tr>
|
|
<tr><td>PVC full</td><td>new pulls fail, cached OK-ish</td><td>mirror 5xx on writes</td><td>yes for missing content</td><td>capacity metrics (proposed)</td><td>availability; GC pressure</td></tr>
|
|
<tr><td>Digest entry GC-evicted</td><td>silent re-fetch on next pull</td><td>mirror -> origin re-sync</td><td>n/a</td><td>upstream egress for "cached" content</td><td>rate-limit exposure returns (section 06)</td></tr>
|
|
<tr><td>Node auth wrong (post-flip)</td><td>ImagePullBackOff</td><td>mirror 401; fallback unreliable on 401</td><td>unreliable</td><td>the canary gate (section 07)</td><td>the outage the gate exists to prevent</td></tr>
|
|
</table></div>
|
|
|
|
<h2 id="digests"><span class="n" aria-hidden="true">05</span>Digests, manifests and the supply chain</h2>
|
|
|
|
<p>Precision matters here, because half the traps on this page come from conflating these
|
|
objects:</p>
|
|
|
|
<ul>
|
|
<li>A <strong>tag</strong> is a mutable, human-readable pointer. Zero, one or many tags can
|
|
reference one manifest.</li>
|
|
<li>A <strong>manifest digest</strong> identifies exact manifest bytes. A multi-platform
|
|
<strong>index</strong> has its own digest; each platform manifest has another; layers and
|
|
config blobs have their own again.</li>
|
|
<li><code>name:tag@digest</code> is client-side grammar, not wire protocol: the OCI
|
|
distribution spec takes a tag <em>or</em> a digest in the URL. Clients resolve the digest
|
|
and ignore the tag (Kubernetes documents exactly this), so digest-pinned pulls are immune
|
|
to tag moves - a fact this page once mislearned as a cache trap and later retested: the
|
|
historical zot rejection of combined references did not reproduce on v2.1.17. Traps are
|
|
perishable; retest your scars.</li>
|
|
<li>Digest pinning provides <strong>content integrity</strong> - not provenance, not
|
|
authorisation, not availability, and not freedom from network fetches on a cold store.</li>
|
|
</ul>
|
|
|
|
<p>For a mirror, digests carry one sharp operational rule, learned here as a crashloop:
|
|
zot converts Docker-schema manifests to OCI by default, and conversion changes the digest -
|
|
silently breaking every digest pin and signature downstream. The pairing that prevents it
|
|
(<code>preserveDigest: true</code> per upstream, with <code>http.compat: ["docker2s2"]</code>)
|
|
is mandatory in both directions: the registry refuses to start with one and not the other.
|
|
This deployment runs both, on all five upstreams - observed in the live config.</p>
|
|
|
|
<p>Preserving digests does <em>not</em> mean the supply chain travels whole. OCI 1.1
|
|
referrers - signatures, SBOMs, attestations attached via the <code>subject</code> field and
|
|
served by the referrers API - are separate objects with their own discovery path. A
|
|
pull-through cache that mirrors manifests and blobs does not automatically carry them.
|
|
Nothing here verifies signatures today; if it did, disconnected verification would also
|
|
need the trust material carried locally (a public key, or a trusted-root bundle for a
|
|
private signing stack). That is stated as a boundary, not an aspiration.</p>
|
|
|
|
<div class="clause"><span class="k">THE HOMELAB CLAUSE</span><br>Production estates promote
|
|
releases by digest between registries - a tag is a suggestion, a digest is a fact - and
|
|
decide explicitly whether referrers travel with them. The lab runs the digest half of that
|
|
discipline; the referrer half is future work and labelled as such.</div>
|
|
|
|
<h2 id="retention"><span class="n" aria-hidden="true">06</span>Retention, GC and the digest-only trap</h2>
|
|
|
|
<div class="scene" id="s23"><div class="inner">
|
|
<img src="__M23__" alt="A stuck intake pod ringed in magenta with three glowing cargo bricks queued behind it - cached content waiting on a jammed process.">
|
|
<div class="s23-pulse" id="pulse"></div>
|
|
<div class="cap">Cached bricks queued behind a jammed intake: availability problems here are quiet, not loud.</div>
|
|
</div></div>
|
|
|
|
<p>This section corrects the largest error in this page's own first edition. I wrote:
|
|
"no retention config means keep everything". That is true for tags, and dangerously false
|
|
for everything else - and "everything else" includes precisely the content this mirror
|
|
exists to hold.</p>
|
|
|
|
<p>The verified semantics, from upstream documentation and the deployed version's source:</p>
|
|
|
|
<ul>
|
|
<li>With no retention configuration, <strong>all tags are retained</strong> - and
|
|
<strong>all untagged manifests are deleted</strong> by garbage collection (unless referenced
|
|
by an index or artifact) once they are older than <code>gcDelay</code>. This deployment's
|
|
delay is the 1-hour default, with GC sweeping daily.</li>
|
|
<li><strong>Digest-only pull-through entries are stored as untagged manifests.</strong>
|
|
That is the exact wording of the upstream fix's problem statement: pull an image through
|
|
the mirror by digest, and the cache entry has no tag to protect it.</li>
|
|
</ul>
|
|
|
|
<div class="sayit"><span class="k">SAY IT</span>A digest-pinned fleet, pulling through a
|
|
mirror whose GC treats digest-only entries as garbage: the cache evicts exactly what the
|
|
estate is built on.</div>
|
|
|
|
<p>Put the pieces together with section 04's revalidation fact and the deployed behaviour
|
|
is this: <em>tag</em> pulls are cached but always phone upstream anyway; <em>digest</em>
|
|
pulls are served locally but their cache entries are eligible for eviction within hours.
|
|
On v2.1.17, this mirror is a rate-limit shield and a latency win - it is <strong>not yet a
|
|
disconnection shelf</strong>, and the first edition of this page was wrong to imply
|
|
otherwise.</p>
|
|
|
|
<div class="diagram"><span class="k">Diagram C · what keeps an object alive</span>
|
|
<svg viewBox="0 0 860 360" role="img" aria-labelledby="dc-t dc-d">
|
|
<title id="dc-t">The OCI object graph and which references protect content from garbage collection</title>
|
|
<desc id="dc-d">A tag points to an index; the index references platform manifests; manifests reference layers and a config blob. Referrers attach to a manifest via a subject field. A separate digest-only manifest sits with no tag pointing at it; it is eligible for garbage collection after the delay on the deployed version. Retention rules that keep tags do not protect the untagged manifest.</desc>
|
|
<defs><marker id="dc-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
|
|
<marker id="dc-b" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#98a2b4"/></marker></defs>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="12" text-anchor="middle">
|
|
<rect x="30" y="30" width="150" height="40" rx="8" fill="none" stroke="#7fe0a7"/><text x="105" y="55" fill="#dfe6f0">tag: v1.2</text>
|
|
<rect x="240" y="30" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="325" y="49" fill="#dfe6f0">index (multi-arch)</text><text x="325" y="64" fill="#98a2b4">own digest</text>
|
|
<rect x="470" y="10" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="555" y="35" fill="#dfe6f0">manifest amd64</text>
|
|
<rect x="470" y="62" width="170" height="40" rx="8" fill="none" stroke="#3fbaf5"/><text x="555" y="87" fill="#dfe6f0">manifest arm64</text>
|
|
<rect x="700" y="10" width="140" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="770" y="35" fill="#dfe6f0">layers + config</text>
|
|
<rect x="700" y="62" width="140" height="40" rx="8" fill="none" stroke="#98a2b4"/><text x="770" y="87" fill="#dfe6f0">layers + config</text>
|
|
<rect x="470" y="130" width="170" height="40" rx="8" fill="none" stroke="#e879f9" stroke-dasharray="4 3"/><text x="555" y="149" fill="#dfe6f0">referrer (sig/SBOM)</text><text x="555" y="164" fill="#e879f9">subject -> manifest</text>
|
|
<line x1="180" y1="50" x2="235" y2="50" stroke="#7fe0a7" marker-end="url(#dc-a)"/>
|
|
<line x1="410" y1="42" x2="465" y2="30" stroke="#3fbaf5" marker-end="url(#dc-a)"/>
|
|
<line x1="410" y1="58" x2="465" y2="80" stroke="#3fbaf5" marker-end="url(#dc-a)"/>
|
|
<line x1="640" y1="30" x2="695" y2="30" stroke="#98a2b4" marker-end="url(#dc-b)"/>
|
|
<line x1="640" y1="82" x2="695" y2="82" stroke="#98a2b4" marker-end="url(#dc-b)"/>
|
|
<line x1="555" y1="130" x2="555" y2="106" stroke="#e879f9" stroke-dasharray="4 3" marker-end="url(#dc-a)"/>
|
|
<text x="105" y="120" fill="#7fe0a7">a tag is a keep-alive</text>
|
|
<rect x="30" y="215" width="280" height="52" rx="8" fill="none" stroke="#e8b44a"/><text x="170" y="236" fill="#dfe6f0">digest-only cached manifest</text><text x="170" y="253" fill="#e8b44a">NO tag references it</text>
|
|
<rect x="380" y="215" width="220" height="52" rx="8" fill="none" stroke="#ff8fa3"/><text x="490" y="236" fill="#dfe6f0">GC after gcDelay (1h here)</text><text x="490" y="253" fill="#ff8fa3">deleteUntagged default: true</text>
|
|
<line x1="310" y1="241" x2="375" y2="241" stroke="#ff8fa3" marker-end="url(#dc-a)"/>
|
|
<text x="430" y="310" fill="#98a2b4">keepTags rules protect TAGS in their repo. Nothing on v2.1.17/18 protects the amber box;</text>
|
|
<text x="430" y="330" fill="#98a2b4">pull-aware keepUntagged ships in v2.1.19 (2026-08-04). Node-local caches can mask the eviction for days.</text>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">green = protected by a tag · amber/red = the eviction path · dashed magenta = referrers (separate lifecycle)</p>
|
|
<p class="why">The lesson: reachability, not existence, is what GC respects - and a
|
|
pull-through cache full of digest pulls is a graveyard of unreachable-by-tag objects.</p></div>
|
|
|
|
<p>Retention policy semantics, stated precisely because two nearby systems use opposite
|
|
rules:</p>
|
|
|
|
<ul>
|
|
<li>Retention policies match per repository, <strong>first match wins</strong> - order the
|
|
list most-specific-first. A repository matching no policy keeps its tags.</li>
|
|
<li>Within a matched repository, one <code>keepTags</code> rule inverts the default:
|
|
non-matching tags in <em>that repository</em> become deletable. The inversion is scoped to
|
|
the repository, not global - the first edition of this page overstated it.</li>
|
|
<li>Untagged manifests are governed separately (<code>deleteUntagged</code>, default true),
|
|
and on the deployed version no retention rule can protect them. Pull-aware
|
|
<code>keepUntagged</code> exists upstream from v2.1.19.</li>
|
|
<li>Authorisation policies (section 07) use <strong>longest match</strong>, not first
|
|
match. Two adjacent config blocks, two opposite precedence rules. Label which one you are
|
|
reasoning about.</li>
|
|
</ul>
|
|
|
|
<p><strong>Mitigations, honestly ranked:</strong> upgrade to v2.1.19+ and configure
|
|
<code>keepUntagged</code> with pull-activity rules (the designed fix); until then, widen
|
|
<code>gcDelay</code>/<code>retention.delay</code> so eviction pressure drops (the upstream
|
|
maintainer's own interim suggestion), or set <code>deleteUntagged: false</code> and accept
|
|
that the cache only grows - a capacity trade, not a free lunch. Disabling GC entirely
|
|
trades eviction for guaranteed storage exhaustion with no reclaim path; it is listed here
|
|
to be argued against.</p>
|
|
|
|
<div class="note warn"><span class="k">SAFE VALIDATION - PROPOSED, NOT YET RUN</span><br>
|
|
The eviction claim above is documented upstream and consistent with this config; it has not
|
|
been reproduced in this lab yet. The safe reproduction, for a <em>disposable</em> zot
|
|
instance only: record version + sanitised config; pull an image by digest through it;
|
|
confirm on the registry side that the stored manifest is untagged; shorten GC timers (in
|
|
the disposable instance only); observe the manifest before and after the sweep; then block
|
|
upstream and repeat the pull from a clean runtime store, recording whether the mirror
|
|
serves or re-fetches. Two warnings from upstream documentation: the retention verification
|
|
tool executes orphan-blob GC for real even in dry-run, and on local storage it requires the
|
|
registry stopped. Never point retention experiments at live storage.</div>
|
|
|
|
<h2 id="auth"><span class="n" aria-hidden="true">07</span>The authentication migration</h2>
|
|
|
|
<div class="scene" id="s22"><div class="inner">
|
|
<img src="__M22__" alt="Four parts left to right: a waiting keyed cartridge, a key wedge, a tall gate frame, and a cartridge beyond the gate with its keyway lit - per-node credentials proven at a checkpoint.">
|
|
<div class="s22-glow" style="left:16%;top:30%" data-band="0"></div>
|
|
<div class="s22-glow" style="left:44%;top:28%" data-band="1"></div>
|
|
<div class="s22-glow" style="left:70%;top:26%" data-band="2"></div>
|
|
<div class="cap">The proving gate: a node counts as migrated when an authenticated pull succeeds where an anonymous one cannot.</div>
|
|
</div></div>
|
|
|
|
<p>Target state: the mirror refuses anonymous pulls. Current state: anonymous read on, with
|
|
per-node credentials staged in machine configs (inert until each node's reboot - the
|
|
documented behaviour). The migration is a sequencing problem with one trap at its centre:</p>
|
|
|
|
<p><strong>While anonymous read is on, an ordinary pull proves nothing about auth.</strong>
|
|
A node whose credentials never applied sends no authorisation header, gets served as an
|
|
anonymous reader, and false-passes the gate. The canary must live in a repository that
|
|
denies anonymous read, so a successful pull can only mean an authenticated pull.</p>
|
|
|
|
<p>That design is now upstream-validated rather than assumed: zot authorisation resolves
|
|
per-repository policies by <strong>longest match</strong> - the most specific path pattern
|
|
wins, and <code>**</code> is explicitly the default policy for anything unmatched. A
|
|
<code>canary/**</code> entry that grants named identities read and carries no
|
|
<code>anonymousPolicy</code> therefore denies anonymous on that path while the glob keeps
|
|
the rest of the shelf open. Maintainer guidance confirms anonymous and authenticated access
|
|
are evaluated independently. The staged policy:</p>
|
|
|
|
<pre>"accessControl": {
|
|
"repositories": {
|
|
"**": {
|
|
"anonymousPolicy": ["read"],
|
|
"policies": [
|
|
{ "users": ["zot-push"], "actions": ["read", "create", "update"] },
|
|
{ "users": ["zot-pull"], "actions": ["read"] }
|
|
]
|
|
},
|
|
"canary/**": {
|
|
"policies": [
|
|
{ "users": ["zot-pull", "zot-push"], "actions": ["read"] }
|
|
]
|
|
}
|
|
}
|
|
}</pre>
|
|
|
|
<div class="note"><span class="k">CLIENT CAVEAT · DOCUMENTED</span><br>Mixed
|
|
anonymous/authenticated policies trigger a Docker-client-specific workaround present in
|
|
this exact version: <code>/v2/</code> returns 401 to Docker user agents so the Docker CLI
|
|
sends credentials, meaning anonymous <em>docker</em> users must log in even for anonymous
|
|
repos. Podman and containerd are unaffected. This estate pulls with containerd, podman and
|
|
buildah, so the caveat is noted rather than felt.</div>
|
|
|
|
<p>The gate, per node, and what each step proves:</p>
|
|
|
|
<ol>
|
|
<li>Stage credentials in the node's machine config (inert; documented as requiring reboot).</li>
|
|
<li>Reboot the node at a planned window.</li>
|
|
<li><strong>Prove identity:</strong> pull the protected canary through the node's own
|
|
runtime. Success = authenticated (anonymous cannot); failure = 401, back to step 1. The
|
|
expected split, to be confirmed against observed responses when the test runs: 401 for
|
|
missing/wrong credentials, 403 for a valid identity lacking the action - upstream docs
|
|
state the 403 case explicitly only for OIDC identities, so the basic-auth 403 boundary is
|
|
listed in the open-verification register rather than asserted.</li>
|
|
<li><strong>Prove interception separately:</strong> the canary pull is a direct reference
|
|
to the mirror, so it cannot prove that <code>docker.io/...</code> references are being
|
|
routed through the mirror at all. That second proof needs an original-upstream reference
|
|
pulled on the node <em>plus</em> the mirror's request log showing it arrive - and a cold
|
|
runtime store, because node-local content will satisfy the pull without any network and
|
|
fake a pass.</li>
|
|
<li>Only when every node passes both proofs does <code>anonymousPolicy</code> come off the
|
|
glob. Rollback is the previous config commit, and it is named before the flip, not after.</li>
|
|
</ol>
|
|
|
|
<div class="diagram"><span class="k">Diagram E · the migration ladder</span>
|
|
<svg viewBox="0 0 860 210" role="img" aria-labelledby="de-t de-d">
|
|
<title id="de-t">The authentication migration ladder with its per-node proof gate and rollback point</title>
|
|
<desc id="de-d">Five stages: anonymous baseline, credentials staged inert, node reboot, the two-part proof - authenticated canary pull plus logged mirror interception - and the final anonymous-off flip with a named rollback commit. A failing node loops from the proof back to staging.</desc>
|
|
<defs><marker id="de-a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#3fbaf5"/></marker>
|
|
<marker id="de-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#e879f9"/></marker></defs>
|
|
<g font-family="ui-monospace,Menlo,monospace" font-size="11" text-anchor="middle">
|
|
<rect x="10" y="70" width="140" height="48" rx="8" fill="none" stroke="#98a2b4"/><text x="80" y="90" fill="#dfe6f0">anon read ON</text><text x="80" y="106" fill="#98a2b4">baseline (today)</text>
|
|
<rect x="180" y="70" width="140" height="48" rx="8" fill="none" stroke="#3fbaf5"/><text x="250" y="90" fill="#dfe6f0">creds staged</text><text x="250" y="106" fill="#98a2b4">inert in config</text>
|
|
<rect x="350" y="70" width="140" height="48" rx="8" fill="none" stroke="#3fbaf5"/><text x="420" y="90" fill="#dfe6f0">node reboots</text><text x="420" y="106" fill="#98a2b4">auth goes live</text>
|
|
<rect x="520" y="70" width="160" height="48" rx="8" fill="none" stroke="#e879f9"/><text x="600" y="86" fill="#dfe6f0">two-part proof</text><text x="600" y="100" fill="#e879f9">canary + logged</text><text x="600" y="113" fill="#e879f9">interception</text>
|
|
<rect x="710" y="70" width="140" height="48" rx="8" fill="none" stroke="#7fe0a7"/><text x="780" y="90" fill="#dfe6f0">anon OFF</text><text x="780" y="106" fill="#98a2b4">rollback = prior commit</text>
|
|
<line x1="150" y1="94" x2="175" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
|
|
<line x1="320" y1="94" x2="345" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
|
|
<line x1="490" y1="94" x2="515" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
|
|
<line x1="680" y1="94" x2="705" y2="94" stroke="#3fbaf5" marker-end="url(#de-a)"/>
|
|
<path d="M600 118 Q 600 175 480 175 L 320 175 Q 250 175 250 122" fill="none" stroke="#e879f9" stroke-dasharray="4 4" marker-end="url(#de-m)"/>
|
|
<text x="440" y="196" fill="#e879f9">401: node loops back - it does not count</text>
|
|
</g>
|
|
</svg>
|
|
<p class="legend">the flip is per-fleet; the proof is per-node - one unproven node under a fleet-wide flip is an outage wearing a green tick</p>
|
|
<p class="why">Why two proofs: identity and interception fail independently, and each has a
|
|
false-pass mode the other cannot detect.</p></div>
|
|
|
|
<p>Behind the gate sit five separate trust domains, deliberately not shared: node pull
|
|
credentials (machine config, reboot-bound), CI push credentials (human-held login), the
|
|
mirror's own upstream credentials (one mounted secret, pod-restart-bound), TLS trust
|
|
(estate wildcard, standard roots), and human admin access (SSO in front of the UI). The
|
|
push credential carries this page's oldest scar: its hash and plaintext once drifted apart,
|
|
rotation became impossible, and pushes stayed dead for twelve days until the rebuilt flow
|
|
landed - three applications shipped the same day it came back. Hash and plaintext now live
|
|
side by side, rotated together by a human, and the automation identity that syncs secrets
|
|
is read-only by design so it can never half-rotate them again.</p>
|
|
|
|
<h2 id="observability"><span class="n" aria-hidden="true">08</span>Observability that means something</h2>
|
|
|
|
<p>"The registry is up" is the least useful sentence in this page. The worst incident in
|
|
this mirror's life: an in-flight sync wedged, every pull from that upstream hung, and the
|
|
health endpoint returned 200 throughout. The log line was almost poetic:</p>
|
|
|
|
<div class="pane-k">Replay · observed 2026-07-07 · the stalled sync</div>
|
|
<pre class="pane">"image already demanded, waiting on channel"
|
|
$ kubectl -n zot rollout restart deploy zot <span class="cm"><- recovery action (not a root cause)</span>
|
|
<span class="cm"># probe an actual manifest, not the process's opinion of itself:</span>
|
|
$ curl -sfS --max-time 10 -o /dev/null -w '%{http_code}\n' \
|
|
https://zot.bztmon.org/v2/library/busybox/manifests/latest \
|
|
-H 'Accept: application/vnd.oci.image.index.v1+json'
|
|
<span class="good">200</span> <span class="cm"><- THIS is "the mirror works"</span></pre>
|
|
|
|
<p>Honesty about that incident, upgraded by reading the source: the log line is
|
|
<em>coalescing by design</em> - concurrent requests for one image join the first sync
|
|
rather than duplicating it, and since the deployed version syncs run on a detached
|
|
background context with a three-hour default timeout, surviving client disconnects.
|
|
Waiters blocking on a genuinely stalled sync until that timeout is consistent with what we
|
|
saw; the restart was a recovery action, and the root cause was never isolated. Two knobs
|
|
this config does not currently set - sync <code>maxRetries</code> (disabled by default
|
|
upstream) and a tighter <code>syncTimeout</code> - are the first things a recurrence should
|
|
change.</p>
|
|
|
|
<p>What this deployment actually exposes, verified against the running version's source:
|
|
<code>/livez</code>, <code>/readyz</code> and <code>/startupz</code> exist as real health
|
|
endpoints (absent from the docs at this version - a documentation gap, not an invention),
|
|
alongside the spec's <code>/v2/</code>. The Prometheus metrics extension exists upstream
|
|
(real series names include <code>zot_http_requests_total</code>,
|
|
<code>zot_repo_storage_bytes</code>, <code>zot_repo_downloads_total</code>,
|
|
<code>zot_storage_lock_latency_seconds</code>) - and is <strong>not enabled in this
|
|
deployment</strong>. Wiring it, with scrape auth, is on the open list; no dashboard here
|
|
pretends otherwise.</p>
|
|
|
|
<p>The layered checks that would actually mean something, each answering one question:</p>
|
|
|
|
<div class="tablewrap"><table>
|
|
<tr><th>Check</th><th>Question it answers</th><th>Status here</th></tr>
|
|
<tr><td><code>/livez</code> / <code>/readyz</code></td><td>is the process up / initialised</td><td><span class="st ok">available</span></td></tr>
|
|
<tr><td>authenticated manifest GET of a local-only canary</td><td>can the right identity read real content from local storage</td><td><span class="st prop">proposed</span></td></tr>
|
|
<tr><td>anonymous GET of the canary expecting 401</td><td>is the protection actually protecting</td><td><span class="st prop">proposed</span></td></tr>
|
|
<tr><td>original-reference pull on a cold node + mirror log line</td><td>is the runtime actually routed through the mirror</td><td><span class="st prop">proposed</span></td></tr>
|
|
<tr><td>digest pull with upstream blocked (disposable env)</td><td>does "cached" mean "served locally"</td><td><span class="st open">open - section 06</span></td></tr>
|
|
<tr><td>PVC usage + growth, GC activity, sync latency</td><td>when does capacity or eviction become the story</td><td><span class="st prop">needs metrics ext</span></td></tr>
|
|
<tr><td>certificate expiry, credential age</td><td>what breaks on a schedule</td><td><span class="st prop">proposed</span></td></tr>
|
|
</table></div>
|
|
|
|
<p>The test matrix for the caching claims - each cell is an experiment, not an assumption
|
|
(status: the two digest rows are documented upstream and consistent with observed
|
|
behaviour here; the disconnected rows have not been run in this lab):</p>
|
|
|
|
<div class="tablewrap"><table>
|
|
<tr><th>Scenario</th><th>Expected on v2.1.17</th><th>What proves it</th></tr>
|
|
<tr><td>cached tag, upstream reachable</td><td>served; upstream contacted anyway (revalidation)</td><td>zot log shows upstream request; latency includes round-trip</td></tr>
|
|
<tr><td>cached tag, upstream unreachable</td><td>fails or stalls until timeout, then local fallback path</td><td>pull timing vs sync timeout; error text</td></tr>
|
|
<tr><td>cached digest, upstream reachable</td><td>served locally, no upstream contact</td><td>absence of upstream request in zot logs during the pull</td></tr>
|
|
<tr><td>cached digest, upstream unreachable</td><td>served locally - IF the entry survived GC</td><td>the section-06 disposable-instance drill</td></tr>
|
|
<tr><td>cold node, warm mirror</td><td>mirror serves; node store fills</td><td>mirror log + no origin egress</td></tr>
|
|
<tr><td>warm node, evicted mirror entry</td><td>pull succeeds from node store - masking the eviction</td><td>this is the false-pass: only mirror-side inspection reveals it</td></tr>
|
|
<tr><td>multi-arch index + child manifests</td><td>index and per-platform manifests are separate cache entries</td><td>per-digest existence checks on the mirror</td></tr>
|
|
</table></div>
|
|
|
|
<h2 id="production"><span class="n" aria-hidden="true">09</span>The production translation</h2>
|
|
|
|
<div class="scene" id="s24"><div class="inner">
|
|
<img src="__M24__" alt="A monolithic vault with a wall of glowing bricks visible through its door, attended by four smaller self-sufficient way-stations - a central release source and site-local mirrors.">
|
|
<div class="cap">A central curated source and site-local registries: each site holds its own shelf, so a cut WAN idles nothing.</div>
|
|
</div></div>
|
|
|
|
<p>The first edition of this section asserted what "production" does. That was the wrong
|
|
register: production chooses from patterns against requirements. The honest version:</p>
|
|
|
|
<ul>
|
|
<li><strong>Single registry, accepted failure domain</strong> - exactly this lab's shape.
|
|
Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or to a paused
|
|
deploy window (fallback off) and that cost is accepted in writing.</li>
|
|
<li><strong>Sync-based HA</strong> - upstream documents active/standby and active/active
|
|
pairs of <em>independent</em> zot instances, each with its own storage, mirroring each
|
|
other behind a load balancer; the documented caveat is the synchronisation window between
|
|
polls, which bounds what a failover can lose.</li>
|
|
<li><strong>Scale-out clustering</strong> - upstream's other documented shape: instances
|
|
shard repositories by hash and proxy to the owner, classically over shared S3-compatible
|
|
storage. That is horizontal scale for load; the shared storage is then the availability
|
|
story, which is a different property than the HA pair above. Naming which property you are
|
|
buying is the design act.</li>
|
|
<li><strong>Edge estates</strong> - a central curated source promotes releases by digest;
|
|
each site runs a small local registry holding its release set, synced on a schedule, so a
|
|
severed WAN idles nothing at pull time. A disconnection claim is only valid when the
|
|
required content, its referrers if policy demands them, trust roots, credentials and the
|
|
registry's own bootstrap image are already local - the checklist from diagram D, applied
|
|
per site.</li>
|
|
</ul>
|
|
|
|
<p>Whatever the topology: recovery time and recovery point get numbers before an incident
|
|
provides them; backups are restore-tested copies off the failure domain (array snapshots
|
|
and RAID protect against disks, not against the array, the site, or an operator error -
|
|
they are inputs to a backup strategy, not the strategy); capacity is planned against the
|
|
retention policy from section 06, because "how big does the cache get" is a policy output,
|
|
not a guess; and the circular dependencies from section 04 are drawn per site, because
|
|
every one of them exists at every scale - the only thing that changes is how expensive
|
|
they are to ignore.</p>
|
|
|
|
<h2 id="verified"><span class="n" aria-hidden="true">10</span>Verified, open, and where the claims come from</h2>
|
|
|
|
<div class="scene" id="s25"><div class="inner">
|
|
<img src="__M25__" alt="A wide flat node slab in an otherwise empty void; a single luminous brick hovers midway in its descent toward the slab - one small transfer of bytes as proof of life.">
|
|
<div class="cap">Proof of life is one authenticated byte transfer through the real path - not a status page.</div>
|
|
</div></div>
|
|
|
|
<p>When a node powers on, this fleet's gate makes it pull one known, first-party image
|
|
through its own runtime before it counts as a member. That gate is the whole page in
|
|
miniature: prove the path, not the process.</p>
|
|
|
|
<div class="tablewrap"><table>
|
|
<caption class="mono" style="text-align:left;color:var(--ink-dim);font-size:.78rem;padding-bottom:.4rem">Architecture decisions</caption>
|
|
<tr><th>Decision</th><th>Chosen</th><th>Why</th><th>Trade-off</th><th>Rollback / alternative</th><th>Status</th></tr>
|
|
<tr><td>Origin fallback</td><td>ON (default kept)</td><td>bootstrap survivability, Loop 1</td><td>silent mirror bypass possible</td><td>skipFallback:true + preseed (airgap shape)</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>Sync mode</td><td>onDemand, all upstreams</td><td>cache follows real usage; Hub-safe</td><td>tag pulls revalidate upstream; digest entries untagged</td><td>polled sync for a curated release set</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>Digest preservation</td><td>preserveDigest + docker2s2, x5</td><td>digest pins + signatures survive mirroring</td><td>mandatory config pairing (crashloop scar)</td><td>none - non-negotiable for digest-pinned estates</td><td><span class="st ok">observed</span></td></tr>
|
|
<tr><td>Namespace layout</td><td>flat (no destinations)</td><td>simplicity at build time</td><td>ordered-trial upstream selection; theoretical collisions</td><td>per-origin destinations + overridePath</td><td><span class="st prop">redesign proposed</span></td></tr>
|
|
<tr><td>Retention</td><td>none configured</td><td>predates understanding the untagged rule</td><td>digest-only entries evict within hours</td><td>v2.1.19+ keepUntagged; interim wider gcDelay</td><td><span class="st open">correction owed</span></td></tr>
|
|
<tr><td>Auth</td><td>anonymous read until per-node proof</td><td>containerd 401-fallback unreliability makes big-bang flips dangerous</td><td>window with no pull auth</td><td>staged flip w/ canary gate + named rollback</td><td><span class="st open">in flight</span></td></tr>
|
|
<tr><td>Metrics</td><td>not enabled</td><td>minimal first deployment</td><td>capacity/eviction invisible</td><td>metrics extension + scrape auth</td><td><span class="st prop">proposed</span></td></tr>
|
|
</table></div>
|
|
|
|
<h3 id="open">Open verification register</h3>
|
|
<ol>
|
|
<li>Digest-entry eviction reproduction in a disposable instance (section 06 procedure).</li>
|
|
<li>Canary repo negative test: anonymous 401 on <code>canary/**</code> while anonymous
|
|
read elsewhere still succeeds; observed 401-vs-403 boundaries recorded.</li>
|
|
<li>Mirror interception proof: original-reference pull on a cold node correlated with the
|
|
mirror's request log.</li>
|
|
<li>Disconnected-serve drill: cached digest pull with upstream blocked, disposable
|
|
environment first.</li>
|
|
<li>Upgrade evaluation: v2.1.19/v2.1.20 for <code>keepUntagged</code>, with the retention
|
|
config written and reviewed before the upgrade, not after.</li>
|
|
<li>Namespace redesign migration plan (per-origin destinations + overridePath), including
|
|
cache re-warm cost.</li>
|
|
</ol>
|
|
|
|
<h3 id="refs">Verified against</h3>
|
|
<ul class="refs">
|
|
<li>zot documentation (versioned): <a href="https://zotregistry.dev/v2.1.18/articles/retention/">retention</a> · <a href="https://zotregistry.dev/v2.1.18/articles/storage/">storage</a> · <a href="https://zotregistry.dev/v2.1.18/articles/mirroring/">mirroring</a> · <a href="https://zotregistry.dev/v2.1.18/articles/authn-authz/">authn/authz</a> · <a href="https://zotregistry.dev/v2.1.18/articles/high-availability/">high availability</a> · <a href="https://zotregistry.dev/v2.1.18/articles/scaleout/">scale-out</a></li>
|
|
<li>zot upstream issues/PRs: <a href="https://github.com/project-zot/zot/issues/4148">4148 digest-only eviction</a> · <a href="https://github.com/project-zot/zot/pull/4191">4191 keepUntagged (v2.1.19)</a> · <a href="https://github.com/project-zot/zot/issues/4128">4128 tag revalidation</a> · <a href="https://github.com/project-zot/zot/issues/4187">4187 containerd ns param</a> · <a href="https://github.com/project-zot/zot/issues/2928">2928 mixed anonymous auth</a> · <a href="https://github.com/project-zot/zot/issues/2505">2505 demand coalescing</a></li>
|
|
<li>Talos: <a href="https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/images-container-runtime/pull-through-cache">pull-through registry cache (v1.13)</a> · containerd: <a href="https://github.com/containerd/containerd/blob/main/docs/hosts.md">registry hosts and the ns parameter</a></li>
|
|
<li>OCI: <a href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md">distribution spec</a> · <a href="https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/">image/distribution 1.1 (referrers)</a> · signing: <a href="https://docs.sigstore.dev/cosign/verifying/verify/">cosign verify</a></li>
|
|
<li>Docker Hub <a href="https://docs.docker.com/docker-hub/usage/">usage and limits</a> · buildah <a href="https://github.com/containers/buildah/blob/main/docs/buildah-build.1.md">build --retry semantics</a></li>
|
|
</ul>
|
|
|
|
<footer>
|
|
DOSSIER / 005 · last validated 2026-08-25 against the running estate and the
|
|
sources above · corrections in this revision: retention/GC semantics, warm-cache
|
|
guarantees, retry arithmetic, upstream namespace routing ·
|
|
<a href="/">The Teaching Lab</a> · <a href="https://git.bztmon.com/jwright/exploded-cluster">source</a>
|
|
</footer>
|
|
|
|
</div>
|
|
|
|
<script src="__MIRROR_JS__"></script>
|