From 7613bfa290e1b0bf18bdd079ae0986b865330c99 Mon Sep 17 00:00:00 2001 From: jwright Date: Mon, 17 Aug 2026 18:57:58 +1000 Subject: [PATCH] second accuracy pass: CVO vs OLM, RHACM push-default, ImageTagMirrorSet, runtime calls CNI, ledger cache/cold-pull nuance, skopeo --all, image ID vs manifest digest, named templates, ascii sweep --- pilot/arc.tpl.html | 63 ++++++++++++++++++++++++-------------- pilot/manifest-docker.json | 2 +- pilot/manifest-node.json | 2 +- pilot/manifest-orbit.json | 2 +- 4 files changed, 43 insertions(+), 26 deletions(-) diff --git a/pilot/arc.tpl.html b/pilot/arc.tpl.html index 8a6a40e..de551e3 100644 --- a/pilot/arc.tpl.html +++ b/pilot/arc.tpl.html @@ -98,8 +98,8 @@

A stack of diffs wearing a name

An image is not a copy of a machine. It is a stack of read-only layers, each recording only what changed from the one beneath. Layers are content-addressed, so an - identical layer is stored once and reused by every image that references it - which is why - the second pull of anything is quick. A base sits at the bottom, your dependencies on it, + identical layer is stored once and reused by every image that references it, so a pull only + fetches the layers you do not already have. A base sits at the bottom, your dependencies on it, your code - usually the smallest layer, always the most volatile - above that.

The top slab in the scene is the odd one out: the writable layer is not part of the image. The runtime adds it when it creates the container, and discards it when @@ -194,8 +194,8 @@ runtime pulls the image (through the mirror of Course VII) and actually creates and starts the container, handing the low-level work to runc or crun. A CNI plugin hands the pod a real IP, and kube-proxy - or an eBPF datapath replacing it - makes Service - addresses route to real pods. The kubelet drives, the runtime executes, the CNI wires, the - proxy routes.

+ addresses route to real pods. The kubelet drives and the runtime executes - and it is the + runtime, not the kubelet, that calls the CNI plugin to wire the pod up.

The kubelet decides what should run. The runtime is what starts it.

Field note. Node NotReady? Check kubelet-to-API-server connectivity before anything else - a node that cannot phone home is presumed lost.

@@ -238,7 +238,8 @@

startup owns warm-up, readiness gates traffic, liveness restarts the truly hung. Confusing them is how healthy pods get executed - a slow start killed by an impatient liveness probe looks exactly like a crash.

-

containerPort is documentation - the app still has to bind the port itself.

+

containerPort is documentation - unless a Service targets it by name, or you + use hostPort. Either way the app still has to bind the port itself.

Field note. Exit 137 is 128 + 9: the process was SIGKILLed. It does not say by whom. The kernel's OOM killer surfaces as reason OOMKilled; a failed liveness probe shows up in the pod's events; eviction and node pressure look different again. @@ -322,8 +323,10 @@ genuinely need privilege get a dedicated ServiceAccount bound to a minimal custom SCC, never the stock one.

The platform runs itself

-

Routes predate Ingress and still rule here. Operators run the platform's own - machinery. The OS underneath is immutable - changed by MachineConfig, never by SSH. +

Routes predate Ingress and still rule here. Operators run machinery on your behalf - + though the split matters: the platform's own operators are driven by the Cluster Version + Operator, while OLM installs and upgrades the add-on Operators you choose from + OperatorHub. The OS underneath is immutable - changed by MachineConfig, never by SSH. And SNO puts the entire promise on one box at the edge. At fleet scale the labels from Course VI decide which of these boxes runs what.

On OpenShift, admission is the interview - the SCC is the dress code.

@@ -369,13 +372,15 @@ git commit; the change history is git log; code review is change control. Git is the record of intent - the cluster's own audit log and the reconciler's sync history still record what actually happened, including everything git never sees.

-

oc apply is for archaeology, not for change.

+

oc is for archaeology. git is for change.

Pull, not push - the security inversion

Here the cluster pulls. Push-based delivery exists and is still GitOps to many - this is the stronger variant, and worth choosing deliberately: no CI system, no laptop, no build pipeline holds a credential that can touch the cluster, because the agent inside holds a read-only deploy key and the trust arrow points out. Compromise the build system and you can - propose a change, which is visible; you cannot reach into production. Hand-edit a live object + propose a change, which is visible; you cannot reach into production through git. It + can still push images - which is the other half of why a manifest should name the digest, not + the tag. Hand-edit a live object and the controller flags it OutOfSync - with selfHeal enabled it puts the object back, and with prune enabled what leaves git leaves the cluster. Both are opt-in: without them the reconciler reports the drift and waits for a human. Rollback is git revert, @@ -387,8 +392,9 @@

Run many OpenShift clusters under a hub - RHACM, with edge clusters arriving through zero-touch provisioning - and nobody applies apps to clusters by hand. Each app carries a Placement that selects cluster labels; the hub matches placements against the - labels a cluster wears, and the chosen cluster pulls its assignment. Labelling the cluster - is the deploy action: attach the label and the app follows, remove it and the app + labels a cluster wears, and the chosen cluster is given its assignment - push-style from the + hub by default, or in a pull model where each cluster runs its own reconciler, which is the + variant this arc teaches. Either way, labelling the cluster is the deploy action: attach the label and the app follows, remove it and the app leaves. The same pull model as above, one level bigger - a cluster's labels are its entitlements, reconciled like everything else.

Label the cluster; the app follows.

@@ -421,8 +427,10 @@

A tag is a promise.
A digest is a fact.

myapp:latest feels like a name. It is a sticky note - a mutable pointer anyone with push rights can peel off one image and press onto another, and - nothing anywhere records that it moved. The digest is the image's actual name: same bytes, - same digest, forever.

+ nothing anywhere records that it moved. The digest - sha256 of the image manifest - is + its actual name: same bytes, same digest, forever. (Careful: the sha256 a build prints is the + local image ID, a different hash from the manifest digest the registry mints on push - the + pushed one is what you pin.)

@@ -452,8 +460,9 @@ upstream once, everything after is local. Rate limits, egress cost, disconnected sites, control - one place to gate and audit what enters. OpenShift formalises the re-route with image mirror rules, and carries a sharp edge: digest-mirror rules rewrite digest pulls - only, so a by-tag pull silently skips the mirror. The pinning habit is what makes the - mirror actually catch the traffic.

+ only, so a by-tag pull silently skips them - unless you also add an + ImageTagMirrorSet, which is the rule type built for tag pulls. Pinning by digest is + still the habit that makes the digest rules catch everything.

At real fleet scale the mirror itself tiers: a central mirror in the cloud fronts upstream once, and every site's mirror pulls from the centre rather than from upstream directly. A new image ripples outward in layers - upstream to the centre, centre to each @@ -462,9 +471,13 @@ leave the building.

Mirrors tier: the load fans out in layers, never all at once.

Build once, promote by copy

-

Every rebuild is a different artefact - different digest, untested by the stages before it. +

Every rebuild is a new artefact - in practice a different digest (reproducible builds + are the deliberate exception), untested by the stages before it. So build once, then promote the same digest through environments by copying, registry - to registry - dev proves the exact bytes prod will run. Human tags ride along; the digest is + to registry - dev proves the exact bytes prod will run. On a multi-arch image that copy needs + --all (and --preserve-digests to fail + loudly rather than quietly), or you copy one architecture and the digest you promoted is not + the digest that lands. Human tags ride along; the digest is the through-line.

If the digest changed, it is not a promotion - it is a new candidate.

Names that can move are convenient exactly until they move. Address content @@ -503,8 +516,9 @@

It's Go under the braces

Helm templates are Go text/template: {{ .Values.device.address }} - is a pipeline walking a values object, _helpers.tpl holds the named - functions every manifest calls. You are not writing YAML - you are writing a program whose + is a pipeline walking a values object, _helpers.tpl holds the + named templates - the partials every manifest includes, not + functions you can call bare. You are not writing YAML - you are writing a program whose output is YAML. So render locally, read the output, and lint what came out, not what went in.

Review the render, not just the template.

Field note. ArgoCD renders charts with @@ -562,8 +576,9 @@ through a read-only deploy key.

Leans on it: sync, rollback, change review, the "who merged this" answer.

-

When it is down: the cluster does not care - the reconciler enforces - the last synced state indefinitely. What stops is change. GitOps degrades to +

When it is down: Kubernetes keeps running the last applied state + indefinitely. The reconciler keeps self-healing only while its rendered manifests are + still cached - hours, not forever, and gone after a restart. What stops is change. GitOps degrades to read-only, which is the graceful half of the design.

@@ -584,8 +599,10 @@

Leans on it: every image pull on every node - boot, reschedule, scale-up, recovery.

When it is down: the sharpest edge. Upstream down + mirror up = nobody - notices. Mirror down on a mirror-only pull path = nothing new schedules, and a rebooting - node may not come back whole.

+ notices, provided the image is already cached - a cold entry still needs upstream. + Mirror down on a mirror-only pull path = nothing new schedules unless the node already + holds the image, and imagePullPolicy: Always turns a mirror + outage into a hard stop.

diff --git a/pilot/manifest-docker.json b/pilot/manifest-docker.json index c214f76..9a14d02 100644 --- a/pilot/manifest-docker.json +++ b/pilot/manifest-docker.json @@ -94,7 +94,7 @@ { "name": "code", "title": "Application code", - "blurb": "Your actual program \u2014 usually the smallest layer, and the one that changes every single build.", + "blurb": "Your actual program - usually the smallest layer, and the one that changes every single build.", "z": 0, "points": [ [ diff --git a/pilot/manifest-node.json b/pilot/manifest-node.json index ea08325..4748eb4 100644 --- a/pilot/manifest-node.json +++ b/pilot/manifest-node.json @@ -257,7 +257,7 @@ { "name": "cni", "title": "CNI plugin", - "blurb": "Hands each pod a real IP. The wiring of the whole machine.", + "blurb": "Hands each pod a real IP - invoked by the container runtime, not by the kubelet.", "z": 0, "points": [ [ diff --git a/pilot/manifest-orbit.json b/pilot/manifest-orbit.json index e60e608..e582c63 100644 --- a/pilot/manifest-orbit.json +++ b/pilot/manifest-orbit.json @@ -319,7 +319,7 @@ { "name": "operators", "title": "Operators", - "blurb": "OLM - the platform runs itself. The rings never stop turning.", + "blurb": "Machinery that runs itself: the Cluster Version Operator drives the platform's own, OLM installs the add-ons you choose.", "z": 0, "points": [ [