second accuracy pass: CVO vs OLM, RHACM push-default, ImageTagMirrorSet, runtime calls CNI, ledger cache/cold-pull nuance, skopeo --all, image ID vs manifest digest, named templates, ascii sweep

pilot/arc.tpl.html

@@ -98,8 +98,8 @@
     <h3>A stack of diffs wearing a name</h3>
     <p>An image is not a copy of a machine. It is a stack of <b>read-only</b> layers, each
       recording only what changed from the one beneath. Layers are content-addressed, so an
-      identical layer is stored once and reused by every image that references it - which is why
+      identical layer is stored once and reused by every image that references it, so a pull only
-      the second pull of anything is quick. A base sits at the bottom, your dependencies on it,
+      fetches the layers you do not already have. A base sits at the bottom, your dependencies on it,
       your code - usually the smallest layer, always the most volatile - above that.</p>
     <p>The top slab in the scene is the odd one out: the <b>writable layer is not part of the
       image</b>. The runtime adds it when it creates the <i>container</i>, and discards it when
@@ -194,8 +194,8 @@
       runtime pulls the image (through the mirror of Course VII) and actually creates and starts
       the container, handing the low-level work to runc or crun. A <b>CNI plugin</b> hands the pod
       a real IP, and <b>kube-proxy</b> - or an eBPF datapath replacing it - makes Service
-      addresses route to real pods. The kubelet drives, the runtime executes, the CNI wires, the
+      addresses route to real pods. The kubelet drives and the runtime executes - and it is the
-      proxy routes.</p>
+      runtime, not the kubelet, that calls the CNI plugin to wire the pod up.</p>
     <p class="sayit">The kubelet decides what should run. The runtime is what starts it.</p>
     <p class="fieldnote"><b>Field note.</b> Node NotReady? Check kubelet-to-API-server
       connectivity before anything else - a node that cannot phone home is presumed lost.</p>
@@ -238,7 +238,8 @@
     <p><b>startup</b> owns warm-up, <b>readiness</b> gates traffic, <b>liveness</b> restarts the
       truly hung. Confusing them is how healthy pods get executed - a slow start killed by an
       impatient liveness probe looks exactly like a crash.</p>
-    <p class="sayit">containerPort is documentation - the app still has to bind the port itself.</p>
+    <p class="sayit">containerPort is documentation - unless a Service targets it by name, or you
+      use hostPort. Either way the app still has to bind the port itself.</p>
     <p class="fieldnote"><b>Field note.</b> Exit 137 is 128 + 9: the process was <b>SIGKILLed</b>.
       It does not say by whom. The kernel's OOM killer surfaces as reason OOMKilled; a failed
       liveness probe shows up in the pod's events; eviction and node pressure look different again.
@@ -322,8 +323,10 @@
       genuinely need privilege get a dedicated ServiceAccount bound to a minimal custom SCC,
       never the stock one.</p>
     <h3>The platform runs itself</h3>
-    <p><b>Routes</b> predate Ingress and still rule here. <b>Operators</b> run the platform's own
+    <p><b>Routes</b> predate Ingress and still rule here. <b>Operators</b> run machinery on your behalf -
-      machinery. The OS underneath is immutable - changed by <b>MachineConfig</b>, never by SSH.
+      though the split matters: the platform's own operators are driven by the Cluster Version
+      Operator, while <b>OLM</b> installs and upgrades the add-on Operators you choose from
+      OperatorHub. The OS underneath is immutable - changed by <b>MachineConfig</b>, never by SSH.
       And <b>SNO</b> puts the entire promise on one box at the edge. At fleet scale the labels
       from Course VI decide which of these boxes runs what.</p>
     <p class="sayit">On OpenShift, admission is the interview - the SCC is the dress code.</p>
@@ -369,13 +372,15 @@
       git commit; the change history is git log; code review <i>is</i> change control. Git is the
       record of <i>intent</i> - the cluster's own audit log and the reconciler's sync history still
       record what actually happened, including everything git never sees.</p>
-    <p class="sayit">oc apply is for archaeology, not for change.</p>
+    <p class="sayit">oc is for archaeology. git is for change.</p>
     <h3>Pull, not push - the security inversion</h3>
     <p>Here the cluster <b>pulls</b>. Push-based delivery exists and is still GitOps to many - this
       is the stronger variant, and worth choosing deliberately: no CI system, no laptop, no build
       pipeline holds a credential that can touch the cluster, because the agent inside holds a
       read-only deploy key and the trust arrow points out. Compromise the build system and you can
-      propose a change, which is visible; you cannot reach into production. Hand-edit a live object
+      propose a change, which is visible; you cannot reach into production <i>through git</i>. It
+      can still push images - which is the other half of why a manifest should name the digest, not
+      the tag. Hand-edit a live object
       and the controller flags it OutOfSync - <b>with selfHeal enabled</b> it puts the object back,
       and <b>with prune enabled</b> what leaves git leaves the cluster. Both are opt-in: without
       them the reconciler reports the drift and waits for a human. Rollback is <span class="mono">git revert</span>,
@@ -387,8 +392,9 @@
     <p>Run many OpenShift clusters under a hub - RHACM, with edge clusters arriving through
       zero-touch provisioning - and nobody applies apps to clusters by hand. Each app carries a
       <b>Placement</b> that selects cluster <b>labels</b>; the hub matches placements against the
-      labels a cluster wears, and the chosen cluster pulls its assignment. Labelling the cluster
+      labels a cluster wears, and the chosen cluster is given its assignment - push-style from the
-      <i>is</i> the deploy action: attach the label and the app follows, remove it and the app
+      hub by default, or in a pull model where each cluster runs its own reconciler, which is the
+      variant this arc teaches. Either way, labelling the cluster <i>is</i> the deploy action: attach the label and the app follows, remove it and the app
       leaves. The same pull model as above, one level bigger - a cluster's labels are its
       entitlements, reconciled like everything else.</p>
     <p class="sayit">Label the cluster; the app follows.</p>
@@ -421,8 +427,10 @@
     <h2 id="c7h" class="thesis">A tag is a promise.<br><em>A digest is a fact.</em></h2>
     <p class="lede"><span class="mono">myapp:latest</span> feels like a name. It is a sticky note - a
       mutable pointer anyone with push rights can peel off one image and press onto another, and
-      nothing anywhere records that it moved. The digest is the image's actual name: same bytes,
+      nothing anywhere records that it moved. The digest - sha256 of the image <i>manifest</i> - is
-      same digest, forever.</p>
+      its actual name: same bytes, same digest, forever. (Careful: the sha256 a build prints is the
+      local image ID, a different hash from the manifest digest the registry mints on push - the
+      pushed one is what you pin.)</p>
   </header>
   <div class="pin">
     <div class="viewport">
@@ -452,8 +460,9 @@
       upstream once, everything after is local. Rate limits, egress cost, disconnected sites,
       control - one place to gate and audit what enters. OpenShift formalises the re-route with
       image mirror rules, and carries a sharp edge: digest-mirror rules rewrite <b>digest pulls
-      only</b>, so a by-tag pull silently skips the mirror. The pinning habit is what makes the
+      only</b>, so a by-tag pull silently skips them - unless you also add an
-      mirror actually catch the traffic.</p>
+      <b>ImageTagMirrorSet</b>, which is the rule type built for tag pulls. Pinning by digest is
+      still the habit that makes the digest rules catch everything.</p>
     <p>At real fleet scale the mirror itself tiers: a <b>central mirror in the cloud</b> fronts
       upstream once, and every site's mirror pulls from the centre rather than from upstream
       directly. A new image ripples outward in layers - upstream to the centre, centre to each
@@ -462,9 +471,13 @@
       leave the building.</p>
     <p class="sayit">Mirrors tier: the load fans out in layers, never all at once.</p>
     <h3>Build once, promote by copy</h3>
-    <p>Every rebuild is a different artefact - different digest, untested by the stages before it.
+    <p>Every rebuild is a <i>new</i> artefact - in practice a different digest (reproducible builds
+      are the deliberate exception), untested by the stages before it.
       So build <b>once</b>, then promote the same digest through environments by copying, registry
-      to registry - dev proves the exact bytes prod will run. Human tags ride along; the digest is
+      to registry - dev proves the exact bytes prod will run. On a multi-arch image that copy needs
+      <span class="mono">--all</span> (and <span class="mono">--preserve-digests</span> to fail
+      loudly rather than quietly), or you copy one architecture and the digest you promoted is not
+      the digest that lands. Human tags ride along; the digest is
       the through-line.</p>
     <p class="sayit">If the digest changed, it is not a promotion - it is a new candidate.</p>
     <p class="closeline">Names that can move are convenient exactly until they move. Address content
@@ -503,8 +516,9 @@
   <div class="prose">
     <h3>It's Go under the braces</h3>
     <p>Helm templates are Go text/template: <span class="mono">{{ .Values.device.address }}</span>
-      is a pipeline walking a values object, <span class="mono">_helpers.tpl</span> holds the named
+      is a pipeline walking a values object, <span class="mono">_helpers.tpl</span> holds the
-      functions every manifest calls. You are not writing YAML - you are writing a program whose
+      <b>named templates</b> - the partials every manifest <span class="mono">include</span>s, not
+      functions you can call bare. You are not writing YAML - you are writing a program whose
       output is YAML. So render locally, read the output, and lint what came out, not what went in.</p>
     <p class="sayit">Review the render, not just the template.</p>
     <p class="fieldnote"><b>Field note.</b> ArgoCD renders charts with
@@ -562,8 +576,9 @@
         through a read-only deploy key.</p>
       <p class="leans"><b>Leans on it:</b> sync, rollback, change review, the "who merged this"
         answer.</p>
-      <p class="down"><b>When it is down:</b> the cluster does not care - the reconciler enforces
+      <p class="down"><b>When it is down:</b> Kubernetes keeps running the last applied state
-        the last synced state indefinitely. What stops is <i>change</i>. GitOps degrades to
+        indefinitely. The reconciler keeps self-healing only while its rendered manifests are
+        still cached - hours, not forever, and gone after a restart. What stops is <i>change</i>. GitOps degrades to
         read-only, which is the graceful half of the design.</p>
     </article>
     <article class="card">
@@ -584,8 +599,10 @@
       <p class="leans"><b>Leans on it:</b> every image pull on every node - boot, reschedule,
         scale-up, recovery.</p>
       <p class="down"><b>When it is down:</b> the sharpest edge. Upstream down + mirror up = nobody
-        notices. Mirror down on a mirror-only pull path = nothing new schedules, and a rebooting
+        notices, <i>provided the image is already cached</i> - a cold entry still needs upstream.
-        node may not come back whole.</p>
+        Mirror down on a mirror-only pull path = nothing new schedules unless the node already
+        holds the image, and <span class="mono">imagePullPolicy: Always</span> turns a mirror
+        outage into a hard stop.</p>
     </article>
   </div>
   <div class="prose">

pilot/manifest-docker.json

@@ -94,7 +94,7 @@
   {
    "name": "code",
    "title": "Application code",
-   "blurb": "Your actual program \u2014 usually the smallest layer, and the one that changes every single build.",
+   "blurb": "Your actual program - usually the smallest layer, and the one that changes every single build.",
    "z": 0,
    "points": [
     [

pilot/manifest-node.json

@@ -257,7 +257,7 @@
   {
    "name": "cni",
    "title": "CNI plugin",
-   "blurb": "Hands each pod a real IP. The wiring of the whole machine.",
+   "blurb": "Hands each pod a real IP - invoked by the container runtime, not by the kubelet.",
    "z": 0,
    "points": [
     [

pilot/manifest-orbit.json

@@ -319,7 +319,7 @@
   {
    "name": "operators",
    "title": "Operators",
-   "blurb": "OLM - the platform runs itself. The rings never stop turning.",
+   "blurb": "Machinery that runs itself: the Cluster Version Operator drives the platform's own, OLM installs the add-ons you choose.",
    "z": 0,
    "points": [
     [