jwright/exploded-cluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

persona round two: namespace/manifest/promise term collisions, acronyms expanded, Operator defined, systemd + host-side proof bridges, requests-vs-limits, third de-slop pass

Browse Source
This commit is contained in:
Jonathon Wright
2026-08-17 19:25:40 +10:00
parent 2e38b27d5f
commit b41564d462
1 changed files with 51 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pilot/arc.tpl.html
+51 -33
View File
@@ -32,9 +32,8 @@
literally. Each course is one machine drawn as a single exploded illustration, sliced into its
real components and wired to your scroll, so the architecture moves while the words explain it.
Start with the aperitif's three terminal commands; finish knowing how a change travels from a
git commit to a running, secret-fed, digest-pinned workload on a fleet. Every claim comes from
running these systems for real, and the library at the end links only to official
documentation.</p>
git commit to a running, secret-fed, digest-pinned workload on a fleet. The library at the end links only to official documentation, so every claim here can be
checked against its source.</p>
</section>
@@ -69,8 +68,9 @@
</div>
<div class="prose">
<p class="aside">A note on the commands: this site uses <span class="mono">oc</span>,
OpenShift's CLI. Everything shown works identically with <span class="mono">kubectl</span> -
<span class="mono">oc</span> is a superset that adds OpenShift's own verbs. Read
OpenShift's CLI. The Kubernetes commands here work the same
under <span class="mono">kubectl</span>; <span class="mono">oc</span> is a superset, and the
OpenShift-only parts (SCCs, Routes) are its own. Read
<span class="mono">kubectl</span> in the docs, type whichever your cluster gives you.</p>
<p>That <span class="mono">1/1</span> reads as containers-ready over containers-wanted: a pod
can hold more than one, which is Course IIIb.</p>
@@ -113,7 +113,15 @@
<p>So what does the image provide? The <b>filesystem that process sees</b>. That is the whole
job, and it is why an image is a stack of layers rather than a disk image.</p>
<h3>What the image actually is</h3>
<p class="fieldnote"><b>Field note.</b> Prove it on your own box.
<span class="mono">podman run -d --name t alpine sleep 300</span>, then on the
<i>host</i>: <span class="mono">ps -ef | grep sleep</span> finds the process,
<span class="mono">lsns -p &lt;pid&gt;</span> lists the namespaces it was handed,
<span class="mono">cat /proc/&lt;pid&gt;/cgroup</span> shows where its limits live, and
<span class="mono">mount | grep overlay</span> shows the layers stitched together. None of
it is exotic. It is your kernel, described differently.</p>
<h3>What an image is made of</h3>
<p>An image is not a copy of a machine. It is a stack of <b>read-only</b> layers, each
recording only what changed from the one beneath. Layers are content-addressed, so an
identical layer is stored once and reused by every image that references it, so a pull only
@@ -188,8 +196,9 @@
authorised and is admitted there, and it is the only component allowed to touch <b>etcd</b> -
the key-value store holding the entire cluster state. Lose etcd and you have lost the
cluster, which is why backing it up is the homework nobody should skip. The <b>scheduler</b>
decides which node a new pod belongs on and writes that decision down; it never starts
anything. The <b>controller-manager</b> runs the reconciliation loops. Nothing talks
decides which node a new pod belongs on - packing by the resources a pod
<i>requests</i>, not by what it currently uses - and writes that decision down; it never
starts anything. The <b>controller-manager</b> runs the reconciliation loops. Nothing talks
sideways - every component watches the api-server.</p>
<p>One of those loops is the chain you will debug most: a <b>Deployment</b> creates a
<b>ReplicaSet</b>, and the ReplicaSet creates <b>pods</b>. That is why the deploy in the
@@ -226,17 +235,19 @@
</div>
</div>
<div class="prose">
<h3>The kubelet decides; the runtime does</h3>
<h3>Where a pod becomes processes</h3>
<p>Every node runs a <b>kubelet</b> - the agent that owns what should be running there. It does
not create containers itself. It speaks <b>CRI</b> to <b>containerd</b> or CRI-O, and that
not create containers itself. It speaks <b>CRI</b> - the Container Runtime Interface - to <b>containerd</b> or CRI-O, and that
runtime pulls the image (through the mirror of Course VII) and actually creates and starts
the container, handing the low-level work to runc or crun. A <b>CNI plugin</b> hands the pod
a real IP, and <b>kube-proxy</b> - or an eBPF datapath replacing it - makes Service
addresses route to real pods - and it is the runtime, not the kubelet, that calls the CNI
plugin to wire the pod up.</p>
the container, handing the low-level work to runc or crun. A <b>CNI plugin</b> (Container Network Interface) hands the pod
a real IP, and <b>kube-proxy</b> - or an eBPF datapath (code running
inside the kernel itself) replacing it - makes Service
addresses route to real pods. One correction worth carrying: the runtime calls the CNI plugin,
not the kubelet.</p>
<p class="sayit">The kubelet decides what should run. The runtime is what starts it.</p>
<p class="fieldnote"><b>Field note.</b> Node NotReady? Check kubelet-to-API-server
connectivity before anything else - a node that cannot phone home is presumed lost.</p>
<p class="fieldnote"><b>Field note.</b> Node NotReady? The kubelet is a systemd unit
like any other: <span class="mono">journalctl -u kubelet</span> on that node, and check it
can still reach the api-server. A node that cannot phone home is presumed lost.</p>
<p class="closeline">The control plane never touches your workload. It writes intent; the
kubelet turns intent into instructions, and the runtime turns instructions into processes. Authority and execution meet exactly here, nowhere
else.</p>
@@ -269,8 +280,8 @@
<h3>The jacket, not the container</h3>
<p>Everything inside the jacket shares a <b>network namespace</b>: one IP, localhost between
friends. Volumes are declared once on the pod, but each container mounts the ones it needs -
sharing storage is opt-in, not automatic. <b>initContainers</b> run first, in order, to completion -
gatekeepers under everything. <b>Sidecars</b> ride along with their own containers and their
sharing storage is opt-in, not automatic. <b>initContainers</b> run first, in order, each to
completion - nothing else in the pod starts until every one of them has exited successfully. <b>Sidecars</b> ride along with their own containers and their
own jobs: proxy, logs, reload.</p>
<h3>Three probes, three different jobs</h3>
<p><b>startup</b> owns warm-up, <b>readiness</b> gates traffic, <b>liveness</b> restarts the
@@ -321,10 +332,12 @@
<p>A <b>Service</b> is the fixed point: a ClusterIP inside, a LoadBalancer at the edge,
Ingress or the Gateway API doing host- and path-routing above. Clients hold the name; the
pods behind it come and go without anyone being told.</p>
<h3>How a Service actually finds its pods</h3>
<h3>How a Service finds its pods</h3>
<p>A Service holds no list of pods. It holds a <b>label selector</b> - match
<span class="mono">app: api</span> - and a controller continuously matches that against every
pod in the namespace, keeping the passing set in an <b>EndpointSlice</b>. Labels are how
pod in the <b>namespace</b> - a Kubernetes namespace this time, a naming boundary for
objects, no relation to the kernel namespaces of Course I - keeping the passing set in an
<b>EndpointSlice</b>. Labels are how
everything here finds everything else, from a Service picking pods to a fleet hub picking
whole clusters (Course VI). Wear the label and you are eligible; readiness decides whether
you stay.</p>
@@ -374,16 +387,18 @@
</div>
<div class="prose">
<h3>The doorman interviews every pod</h3>
<p>The <b>SCC</b> is admission deciding what a pod may BE. The default, restricted-v2, runs
<p>The <b>SCC</b> - Security Context Constraint - is admission deciding what a pod may BE, checked
by the api-server when the pod is created and before any node sees it. The default, restricted-v2, runs
your container as a <b>random non-root UID</b> - your image has to cope. Workloads that
genuinely need privilege get a dedicated ServiceAccount bound to a minimal custom SCC,
never the stock one.</p>
<h3>The platform runs itself</h3>
<p><b>Routes</b> predate Ingress and still rule here. <b>Operators</b> run machinery on your behalf -
though the split matters: the platform's own operators are driven by the Cluster Version
<p><b>Routes</b> predate Ingress and still rule here. An <b>Operator</b> is a controller paired with a custom
resource: you describe what you want in YAML, and its controller builds it and keeps it true
- the reconciliation loop of C-II, sold as a product. The split matters: the platform's own operators are driven by the Cluster Version
Operator, while <b>OLM</b> installs and upgrades the add-on Operators you choose from
OperatorHub. The OS underneath is immutable - changed by <b>MachineConfig</b>, never by SSH.
And <b>SNO</b> puts the entire promise on one box at the edge. At fleet scale the labels
And <b>SNO</b> - single-node OpenShift - puts the whole cluster on one box at the edge. At fleet scale the labels
from Course VI decide which of these boxes runs what.</p>
<p class="sayit">On OpenShift, admission is the interview - the SCC is the dress code.</p>
<p class="fieldnote"><b>Field note.</b> Deployment stuck at 0/1 with no pod at all? The
@@ -440,12 +455,13 @@
and <b>with prune enabled</b> what leaves git leaves the cluster. Both are opt-in: without
them the reconciler reports the drift and waits for a human. Rollback is <span class="mono">git revert</span>,
which is why commit hygiene is an operational skill.</p>
<p class="fieldnote"><b>Field note.</b> Fighting the reconciler over SSH is arm-wrestling a
machine that does not get tired - C-II's warning, now with a face. The fix is always
upstream, in git.</p>
<p class="fieldnote"><b>Field note.</b> A hand-patch on a live object survives exactly until
the next sync. If you must touch production directly to stop the bleeding, open the pull
request in the same hour - otherwise the reconciler quietly undoes your fix, and the outage
returns with nobody able to say why.</p>
<h3>At fleet scale, the label is the deploy button</h3>
<p>Run many OpenShift clusters under a hub - RHACM, with edge clusters arriving through
zero-touch provisioning - and nobody applies apps to clusters by hand. Each app carries a
<p>Run many OpenShift clusters under a hub - RHACM, Red Hat's fleet manager, with edge clusters
arriving through zero-touch provisioning - and nobody applies apps to clusters by hand. Each app carries a
<b>Placement</b> that selects cluster <b>labels</b>; the hub matches placements against the
labels a cluster wears, and the chosen cluster is given its assignment - push-style from the
hub by default, or in a pull model where each cluster runs its own reconciler, which is the
@@ -482,7 +498,9 @@
<h2 id="c7h" class="thesis">A tag is a promise.<br><em>A digest is a fact.</em></h2>
<p class="lede"><span class="mono">myapp:latest</span> feels like a name. It is a sticky note - a
mutable pointer anyone with push rights can peel off one image and press onto another, and
nothing anywhere records that it moved. The digest - sha256 of the image <i>manifest</i> - is
nothing anywhere records that it moved. The digest - sha256 of the image <i>manifest</i>, the
small JSON index listing an image's layers, and no relation to the YAML manifests you apply
to a cluster - is
its actual name: same bytes, same digest, forever. (Careful: the sha256 a build prints is the
local image ID, a different hash from the manifest digest the registry mints on push - the
pushed one is what you pin.)</p>
@@ -501,7 +519,7 @@
<div class="prose">
<h3>Say the name properly</h3>
<p>Three ways to name an image, in rising order of honesty: <span class="mono">:latest</span>
(a moving target), <span class="mono">:1.4.2</span> (a promise somebody keeps, until they
(a moving target), <span class="mono">:1.4.2</span> (a label someone maintains, until they
re-push it), and <span class="mono">name:1.4.2@sha256:...</span> - a fact. The tag stays for
human eyes; the digest does the pulling. Pin by digest and "what is running?" has exactly one
answer.</p>
@@ -529,8 +547,8 @@
<p>Every rebuild is a <i>new</i> artefact - in practice a different digest (reproducible builds
are the deliberate exception), untested by the stages before it.
So build <b>once</b>, then promote the same digest through environments by copying, registry
to registry - dev proves the exact bytes prod will run. On a multi-arch image that copy needs
<span class="mono">--all</span> (and <span class="mono">--preserve-digests</span> to fail
to registry - dev proves the exact bytes prod will run. On a multi-arch image that copy
(<span class="mono">skopeo copy</span>) needs <span class="mono">--all</span> (and <span class="mono">--preserve-digests</span> to fail
loudly rather than quietly), or you copy one architecture and the digest you promoted is not
the digest that lands. Human tags ride along; the digest is
the through-line.</p>