jwright/exploded-cluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

voice: strip mid-paragraph bold term-emphasis (123 -> 43 <b>, keeping only structural labels + library glossary) and cut dash density (121 -> 89, no paragraph now stacks 3+ dashes) - reads authored, not generated

Browse Source
This commit is contained in:
Jonathon Wright
2026-08-17 20:15:45 +10:00
parent 26275c6262
commit 20b1dcfb9a
1 changed files with 99 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pilot/arc.tpl.html
+99 -100
View File
@@ -10,25 +10,25 @@
<nav class="toc" aria-label="Course index">
<p class="kt">Index</p>
<ol>
<li><a href="#c0"><b>Course 00</b><span>Aperitif - three delivery actions and one check</span></a></li>
<li><a href="#c1"><b>Course I</b><span>Podman - an image is a stack of frozen diffs</span></a></li>
<li><a href="#c2"><b>Course II</b><span>Kubernetes - a cluster is a promise, not a place</span></a></li>
<li><a href="#c3a"><b>Course IIIa</b><span>The node - where intent becomes a process</span></a></li>
<li><a href="#c3b"><b>Course IIIb</b><span>The pod - one IP, shared fate</span></a></li>
<li><a href="#c4"><b>Course IV</b><span>The traffic - pods die constantly, the address does not</span></a></li>
<li><a href="#c5"><b>Course V</b><span>OpenShift - Kubernetes with opinions, and a security guard</span></a></li>
<li><a href="#c6"><b>Course VI</b><span>GitOps - nobody deploys anything; the cluster syncs itself</span></a></li>
<li><a href="#c7"><b>Course VII</b><span>The image supply chain - a tag is a promise, a digest is a fact</span></a></li>
<li><a href="#c8"><b>Course VIII</b><span>Helm - a chart is a function; the values file is the cluster speaking</span></a></li>
<li><a href="#apx"><b>Appendix</b><span>The dependency ledger - GitHub, Azure Key Vault, zot</span></a></li>
<li><a href="#flow"><b>Blueprint</b><span>The delivery flow, end to end</span></a></li>
<li><a href="#library"><b>Library</b><span>Official documentation for every technology on this page</span></a></li>
<li><a href="#c0">Course 00<span>Aperitif - three delivery actions and one check</span></a></li>
<li><a href="#c1">Course I<span>Podman - an image is a stack of frozen diffs</span></a></li>
<li><a href="#c2">Course II<span>Kubernetes - a cluster is a promise, not a place</span></a></li>
<li><a href="#c3a">Course IIIa<span>The node - where intent becomes a process</span></a></li>
<li><a href="#c3b">Course IIIb<span>The pod - one IP, shared fate</span></a></li>
<li><a href="#c4">Course IV<span>The traffic - pods die constantly, the address does not</span></a></li>
<li><a href="#c5">Course V<span>OpenShift - Kubernetes with opinions, and a security guard</span></a></li>
<li><a href="#c6">Course VI<span>GitOps - nobody deploys anything; the cluster syncs itself</span></a></li>
<li><a href="#c7">Course VII<span>The image supply chain - a tag is a promise, a digest is a fact</span></a></li>
<li><a href="#c8">Course VIII<span>Helm - a chart is a function; the values file is the cluster speaking</span></a></li>
<li><a href="#apx">Appendix<span>The dependency ledger - GitHub, Azure Key Vault, zot</span></a></li>
<li><a href="#flow">Blueprint<span>The delivery flow, end to end</span></a></li>
<li><a href="#library">Library<span>Official documentation for every technology on this page</span></a></li>
</ol>
</nav>
<section class="purpose" aria-labelledby="purph">
<h2 class="kt" id="purph">What this teaches</h2>
<p>The Exploded Cluster teaches how modern container platforms work by taking them apart -
<p>The Exploded Cluster teaches how modern container platforms work by taking them apart,
literally. Each course is one machine drawn as a single exploded illustration, sliced into its
real components and wired to your scroll, so the architecture moves while the words explain it.
Start with the aperitif&#39;s four terminal commands - three that deliver, one that checks - and
@@ -74,7 +74,7 @@
<span class="mono">kubectl</span> in the docs, type whichever your cluster gives you.</p>
<p>That <span class="mono">1/1</span> reads as containers-ready over containers-wanted: a pod
can hold more than one, which is Course IIIb.</p>
<p>Three of those were delivery actions - <b>build</b>, <b>push</b>, <b>apply</b> - and the
<p>Three of those were delivery actions - build, push, apply, and the
last, <span class="mono">oc get pods</span>, only checked the result. Built of what, exactly?
Pushed to where, and what travelled? Applied, which is not the same as launched. Running,
according to whom? Each course below takes one of those words apart.</p>
@@ -87,7 +87,7 @@
<p class="eyebrow">Course I &middot; Podman</p>
<h2 id="c1h" class="thesis">An image is not a box.<br><em>It is a stack of frozen diffs.</em></h2>
<p class="lede">Scroll, and the thing you keep calling "a container image" comes apart in your
hands. Four layers. Each one only stores what changed from the layer under it - and here they
hands. Four layers. Each one only stores what changed from the layer under it, and here they
rise one at a time, bottom up.</p>
</header>
<div class="pin">
@@ -104,14 +104,14 @@
<div class="prose">
<h3>A container is a process, not a machine</h3>
<p>If you arrived here from Linux, take this translation first: a running container is an
<b>ordinary process on your kernel</b>. No guest OS, no hypervisor. The kernel gives it
<b>namespaces</b> so it sees its own PID tree, mounts, network and hostname, and
<b>cgroups</b> so its CPU and memory can be capped. <span class="mono">ps</span> on the host
ordinary process on your kernel. No guest OS, no hypervisor. The kernel gives it
namespaces so it sees its own PID tree, mounts, network and hostname, and
cgroups so its CPU and memory can be capped. <span class="mono">ps</span> on the host
lists it. <span class="mono">kill</span> on the host kills it. Podman leans into that: no
daemon sits in the middle - the container is a child of your own shell - and rootless mode
daemon sits in the middle - the container is a child of your own shell, and rootless mode
maps your user onto root <i>inside</i> the container through a user namespace, so root in
there is an unprivileged UID out here.</p>
<p>So what does the image provide? The <b>filesystem that process sees</b>. That is the whole
<p>So what does the image provide? The filesystem that process sees. That is the whole
job, and it is why an image is a stack of layers rather than a disk image.</p>
<p class="fieldnote"><b>Field note.</b> Prove it on your own box.
@@ -123,19 +123,19 @@
it is exotic. It is your kernel, described differently.</p>
<h3>What an image is made of</h3>
<p>An image is not a copy of a machine. It is a stack of <b>read-only</b> layers, each
<p>An image is not a copy of a machine. It is a stack of read-only layers, each
recording only what changed from the one beneath. Layers are content-addressed, so an
identical layer is stored once and reused by every image that references it, so a pull
fetches just the layers you do not already have. A base sits at the bottom, your dependencies on it,
your code - usually the smallest layer, and typically the most volatile - above that.</p>
<p>Those layers become one filesystem through a <b>union mount</b> - overlayfs, the same
<p>Those layers become one filesystem through a union mount - overlayfs, the same
kernel feature you can mount by hand. The read-only image layers are the lower dirs; the
container gets a fresh <b>upper dir</b> of its own. Writes land in the upper, and editing an
container gets a fresh upper dir of its own. Writes land in the upper, and editing an
existing file copies it up there first, leaving the image layer untouched underneath.</p>
<p>That upper dir is the top slab in the scene, and it is the odd one out: the <b>writable
layer is not part of the image</b>. The runtime creates it with the <i>container</i> and
<p>That upper dir is the top slab in the scene, and it is the odd one out: the writable
layer is not part of the image. The runtime creates it with the <i>container</i> and
discards it when that container dies. Nothing written there ships, and nothing written there
survives - which is the entire reason volumes exist.</p>
survives, which is the entire reason volumes exist.</p>
<p class="sayit">Change a layer and every layer above it must be rebuilt.</p>
<h3>The order is a caching decision</h3>
<p>The builder caches layer by layer, and a cached layer survives only while everything
@@ -160,7 +160,7 @@
<p class="eyebrow">Course II &middot; Kubernetes</p>
<h2 id="c2h" class="thesis">A cluster is a promise,<br><em>not a place.</em></h2>
<p class="lede">You never tell Kubernetes <i>how</i> to run your app. You describe what you
want - declarative intent - and the cluster works continuously to make it true. Scroll, and
want - declarative intent, and the cluster works continuously to make it true. Scroll, and
the formation splits: the half that decides rises, the halves that run spread below.</p>
</header>
<div class="pin">
@@ -176,25 +176,25 @@
</div>
<div class="prose">
<h3>Desired against actual, on a loop</h3>
<p>The habit underneath everything: the <b>reconciliation loop</b> - compare desired state
<p>The habit underneath everything: the reconciliation loop - compare desired state
against actual state, fix the difference, repeat. That <span class="mono">oc apply</span>
didn't launch anything; it filed paperwork. The machine took it from there, and it keeps taking it from there: kill a pod and it returns, not because something noticed the
crash but because the loop noticed the difference.</p>
<p class="sayit">Kubernetes doesn't run your app - it reconciles it.</p>
<h3>One half decides, one half runs</h3>
<p>The split in the scene is the split that makes everything else possible: a <b>control
plane</b> that decides - holds the truth, schedules, reconciles - and <b>worker nodes</b>
<p>The split in the scene is the split that makes everything else possible: a control
plane that decides - holds the truth, schedules, reconciles, and worker nodes
that run pods. The workers are deliberately interchangeable: identical, replaceable, cattle
from day one. Authority does not live where the work happens.</p>
<p class="fieldnote"><b>Field note.</b> If you SSH into a node to "fix" a workload, a
controller may reconcile your change away. Change the desired state instead, and let the loop
carry it.</p>
<h3>What the control plane is made of</h3>
<p>The core control plane is the <b>API server</b>, <b>scheduler</b> and
<b>controller-manager</b>, backed by <b>etcd</b>; cloud deployments may also run a
<p>The core control plane is the API server, scheduler and
controller-manager, backed by etcd; cloud deployments may also run a
cloud-controller-manager. Naming them makes the later error messages readable. The API server
(<span class="mono">kube-apiserver</span>) is the only door: requests authenticate, are
authorised and are admitted there, and it is the one component that talks to <b>etcd</b>, the
authorised and are admitted there, and it is the one component that talks to etcd, the
key-value store holding Kubernetes API state. Lose etcd and you have lost the cluster, which
is why backing it up is the homework to not skip. The scheduler decides which node a new pod
belongs on - packing by the resources a pod <i>requests</i>, not by what it currently uses -
@@ -202,10 +202,10 @@
reconciliation loops. Control-plane state converges through the API server: the scheduler and
built-in controllers watch API objects and write decisions back through the API rather than
modifying etcd directly.</p>
<p>One of those loops is the chain you will debug most: a <b>Deployment</b> creates a
<b>ReplicaSet</b>, and the ReplicaSet creates <b>pods</b>. That is why the deploy in the
<p>One of those loops is the chain you will debug most: a Deployment creates a
ReplicaSet, and the ReplicaSet creates pods. That is why the deploy in the
aperitif printed <span class="mono">deployment.apps/api created</span> and you then went
looking for a pod - and why, when a rollout is stuck with no pod at all, the answer is
looking for a pod, and why, when a rollout is stuck with no pod at all, the answer is
upstream in that chain rather than on any node.</p>
<p class="sayit">etcd is the durable store for Kubernetes API state; controllers reconcile from it continually.</p>
<p class="closeline">Everything the delivery arc teaches from Course VI onward is the same
@@ -222,14 +222,14 @@
<p class="eyebrow">Course IIIa &middot; The node</p>
<h2 id="c3ah" class="thesis">Where intent<br><em>becomes a process.</em></h2>
<p class="lede">Everything so far was decision. This is the machine where a pod stops being
paperwork and starts being a process - and the chain assembles link by link as you scroll.</p>
paperwork and starts being a process, and the chain assembles link by link as you scroll.</p>
</header>
<div class="pin">
<div class="viewport">
<figure class="scene">
<img class="flat" src="__HERO_NODE__"
alt="A diagonal chain of node machinery: a visor-lit kubelet module, a layered
container-runtime engine, a ported CNI ring, a magenta routing prism - and a
container-runtime engine, a ported CNI ring, a magenta routing prism, and a
small glowing pod capsule descending toward the engine.">
</figure>
<ol class="legend"></ol>
@@ -237,13 +237,13 @@
</div>
<div class="prose">
<h3>Where a pod becomes processes</h3>
<p>The <b>scheduler</b> decides which node a pod belongs on; the <b>kubelet</b> on that node
makes the assigned PodSpec real; the <b>runtime</b> creates and starts the containers. The
kubelet does not place pods, and does not itself create containers - it speaks <b>CRI</b> (the
Container Runtime Interface) to <b>containerd</b> or CRI-O, and that runtime pulls the image
<p>The scheduler decides which node a pod belongs on; the kubelet on that node
makes the assigned PodSpec real; the runtime creates and starts the containers. The
kubelet does not place pods, and does not itself create containers - it speaks CRI (the
Container Runtime Interface) to containerd or CRI-O, and that runtime pulls the image
(through the mirror of Course VII) and starts it, handing the low-level work to runc or crun.
A <b>CNI plugin</b> (Container Network Interface) gives the pod a real IP - called by the
runtime, not the kubelet - and <b>kube-proxy</b>, or an eBPF datapath replacing it, makes
A CNI plugin (Container Network Interface) gives the pod a real IP - called by the
runtime, not the kubelet, and kube-proxy, or an eBPF datapath replacing it, makes
Service addresses route to real pods.</p>
<p class="sayit">The scheduler places the pod. The kubelet realises it. The runtime runs it.</p>
<p class="fieldnote"><b>Field note.</b> Node NotReady? The kubelet is a systemd unit
@@ -279,26 +279,26 @@
</div>
<div class="prose">
<h3>The jacket, not the container</h3>
<p>Everything inside the jacket shares a <b>network namespace</b>: one IP, localhost between
<p>Everything inside the jacket shares a network namespace: one IP, localhost between
friends. Volumes are declared once on the pod, but each container mounts the ones it needs -
sharing storage is opt-in, not automatic. regular <b>init containers</b> run in order and
complete before the app containers start. <b>Native sidecars</b> are the deliberate exception:
sharing storage is opt-in, not automatic. regular init containers run in order and
complete before the app containers start. Native sidecars are the deliberate exception:
restartable init containers that keep running alongside the app, for jobs like proxy, logs and
config reload.</p>
<h3>Three probes, three different jobs</h3>
<p><b>startup</b> owns warm-up, <b>readiness</b> gates traffic, <b>liveness</b> restarts the
<p>startup owns warm-up, readiness gates traffic, liveness restarts the
truly hung. Confusing them is how healthy pods get executed - a slow start killed by an
impatient liveness probe looks exactly like a crash.</p>
<p class="sayit">containerPort is documentation - unless a Service targets it by name, or you
use hostPort. Either way the app still has to bind the port itself.</p>
<p class="fieldnote"><b>Field note.</b> Exit 137 is 128 + 9: the process was <b>SIGKILLed</b>.
<p class="fieldnote"><b>Field note.</b> Exit 137 is 128 + 9: the process was SIGKILLed.
It does not say by whom. The kernel's OOM killer surfaces as reason OOMKilled; a failed
liveness probe shows up in the pod's events; eviction and node pressure look different again.
One exit code, several possible crimes - read the termination reason and the events, never
the number alone: <span class="mono">oc describe pod &lt;name&gt;</span> shows both together,
and <span class="mono">oc logs --previous</span> shows what the dead container said on its
way out. Worth connecting to what you already know: a memory limit becomes a
<b>cgroup ceiling</b>, and the kernel's OOM killer enforces it just as it would for any
cgroup ceiling, and the kernel's OOM killer enforces it just as it would for any
other process on the box.</p>
<p class="closeline">The pod is the smallest schedulable unit - the jacket, never the
container. Once that distinction lands, half of Kubernetes networking stops being
@@ -314,7 +314,7 @@
<header class="chead">
<p class="eyebrow">Course IV &middot; The traffic</p>
<h2 id="c4h" class="thesis">Pods die constantly.<br><em>The address does not.</em></h2>
<p class="lede">Pods die, respawn and change addresses - and traffic still arrives. Scroll,
<p class="lede">Pods die, respawn and change addresses, and traffic still arrives. Scroll,
and the delivery route assembles checkpoint by checkpoint; watch what happens to the pod
that stops answering.</p>
</header>
@@ -324,36 +324,35 @@
<img class="flat" src="__HERO_TRAFFIC__"
alt="A delivery route in the void: a glowing client orb, a fanned load-balancer
wedge, an open ingress doorframe, a Service prism with a bright core, a lit
ready pod - and a dark unlit pod fallen out of the line.">
ready pod, and a dark unlit pod fallen out of the line.">
</figure>
<ol class="legend"></ol>
</div>
</div>
<div class="prose">
<h3>The stable name in front of the churn</h3>
<p>A <b>Service</b> is the fixed point: a ClusterIP inside, a LoadBalancer at the edge,
<p>A Service is the fixed point: a ClusterIP inside, a LoadBalancer at the edge,
Ingress or the Gateway API doing host- and path-routing above. Clients hold the name; the
pods behind it come and go without anyone being told.</p>
<h3>How a Service finds its pods</h3>
<p>A Service holds no list of pods. It holds a <b>label selector</b> - match
<span class="mono">app: api</span> - and a controller continuously matches that against every
pod in the <b>namespace</b> - a Kubernetes namespace this time, a naming boundary for
objects, no relation to the kernel namespaces of Course I - keeping the passing set in an
<b>EndpointSlice</b>. Labels are how
<p>A Service holds no list of pods. It holds a label selector (say
<span class="mono">app: api</span>), and a controller continuously matches that against every
pod in the namespace. That is a Kubernetes namespace, a naming boundary for objects, unrelated
to the kernel namespaces of Course I; the passing set lands in an EndpointSlice. Labels are how
everything here finds everything else, from a Service picking pods to a fleet hub picking
whole clusters (Course VI). Wear the label and you are eligible; readiness decides whether
you stay.</p>
<p>A <b>ClusterIP</b> is a virtual Service address implemented by the node dataplane rather
<p>A ClusterIP is a virtual Service address implemented by the node dataplane rather
than an application process listening on that IP. In iptables or nftables mode,
<b>kube-proxy</b> installs rules that steer Service traffic to endpoint IPs; if you have
kube-proxy installs rules that steer Service traffic to endpoint IPs; if you have
written a DNAT rule by hand, that mode will feel familiar. In IPVS mode, kube-proxy binds
Service IPs to the <span class="mono">kube-ipvs0</span> dummy interface and creates IPVS
virtual servers. eBPF implementations can replace kube-proxy with their own dataplane. Cluster
DNS (<b>CoreDNS</b>) resolves <span class="mono">api.myns.svc.cluster.local</span> to the
DNS (CoreDNS) resolves <span class="mono">api.myns.svc.cluster.local</span> to the
Service address.</p>
<h3>Readiness decides membership</h3>
<p>When a configured readiness probe fails, the pod becomes <b>unready</b> and normal
<p>When a configured readiness probe fails, the pod becomes unready and normal
Kubernetes Service traffic stops selecting it for new connections. There is no error at the
client - the pod drops out of the endpoint set. That is the feature: broken instances take
themselves out of rotation. It is also the first place to look when traffic "disappears".</p>
@@ -363,7 +362,7 @@
the only address worth knowing.</p>
<p class="sayit">For ordinary Service traffic: address the Service, not a pod.</p>
<p class="fieldnote"><b>Field note.</b> "The network is broken" after a deploy is usually
readiness telling the truth about your app - not the network lying about your packets.</p>
readiness telling the truth about your app, not the network lying about your packets.</p>
<p class="closeline">The dark pod in the scene is not an error state - it is the system
working. For pods with readiness probes, eligibility for Service traffic is continually
re-evaluated for the life of the pod.</p>
@@ -393,20 +392,20 @@
</div>
<div class="prose">
<h3>The doorman interviews every pod</h3>
<p>The <b>SCC</b> - Security Context Constraint - is admission deciding what a pod may BE, checked
<p>The SCC - Security Context Constraint - is admission deciding what a pod may BE, checked
by the api-server when the pod is created and before any node sees it. Under OpenShift's restricted
SCCs, workloads normally run as a <b>non-root UID drawn from the project's allocated range</b>,
SCCs, workloads normally run as a non-root UID drawn from the project's allocated range,
so an image has to work with an arbitrary permitted UID rather than assuming a fixed user.
Workloads that genuinely need privilege get a dedicated ServiceAccount bound to the minimum
SCC that grants it - not the stock one.</p>
SCC that grants it, not the stock one.</p>
<h3>The platform runs itself</h3>
<p><b>Routes</b> predate Ingress and still rule here. An <b>Operator</b> is a controller paired with a
<p>Routes predate Ingress and still rule here. An Operator is a controller paired with a
custom resource: you describe what you want in YAML, and its controller builds it and keeps it
true - an Operator applies the same reconciliation pattern to an application or platform
true. An Operator applies the same reconciliation pattern to an application or platform
capability. The split matters: the platform's own operators are driven by the Cluster Version
Operator, while <b>OLM</b> installs and upgrades the add-on Operators you choose from
OperatorHub. <b>MachineConfig</b> is the supported declarative path for node OS configuration;
ad-hoc SSH changes create drift and are not the intended operating model. And <b>SNO</b> -
Operator, while OLM installs and upgrades the add-on Operators you choose from
OperatorHub. MachineConfig is the supported declarative path for node OS configuration;
ad-hoc SSH changes create drift and are not the intended operating model. And SNO -
single-node OpenShift - puts a whole cluster on one box at the edge. At fleet scale the labels
from Course VI decide which of these boxes runs what.</p>
<p class="sayit">On OpenShift, admission is the interview - the SCC is the dress code.</p>
@@ -445,22 +444,22 @@
<div class="prose">
<h3>The loop you already know, one level up</h3>
<p>C-II taught the reconciliation loop: desired versus actual, fix the difference, repeat.
<b>Argo CD</b> applies the same habit to <b>delivery</b>. An Application names a repo, a path
and a revision - watch this branch of this repository - and the controller renders what it
Argo CD applies the same habit to delivery. An Application names a repo, a path
and a revision - watch this branch of this repository, and the controller renders what it
finds there, compares it against the live cluster, and syncs the difference. The deploy button
is a Git commit; the change history is <span class="mono">git log</span>; code review <i>is</i>
change control. Git records <i>intent</i> - the cluster's own audit log and Argo CD's sync
history record what actually happened, including what Git does not see.</p>
<p class="sayit">oc reads the running system; Git records the change.</p>
<h3>Pull, not push - the security inversion</h3>
<p>Here the cluster <b>pulls</b>. Push-based delivery exists and is still GitOps to many; the
<p>Here the cluster pulls. Push-based delivery exists and is still GitOps to many; the
pull variant is worth choosing deliberately, because no CI system, laptop or build pipeline
holds a credential that can touch the cluster - the agent inside holds a read-only deploy key
and the trust arrow points out. Compromise the build system and you can propose a change,
which is visible; you cannot reach into production <i>through Git</i>. It can still push
images, which is the other half of why a manifest should name the digest, not the tag.
Hand-edit a live object and the controller flags it OutOfSync. <b>With self-heal enabled</b>,
Argo CD can restore the desired state; <b>with prune enabled</b>, objects removed from Git can
Hand-edit a live object and the controller flags it OutOfSync. With self-heal enabled,
Argo CD can restore the desired state; with prune enabled, objects removed from Git can
be deleted. Both are opt-in: without them the reconciler reports the drift and waits for a
sync. Rollback is <span class="mono">git revert</span>, which is why commit hygiene is an
operational skill.</p>
@@ -470,8 +469,8 @@
go missing.</p>
<h3>At fleet scale, the label is the deploy button</h3>
<p>Run many OpenShift clusters under a hub - RHACM, Red Hat's fleet manager, with edge clusters
arriving through zero-touch provisioning - and nobody applies apps to clusters by hand. Each app carries a
<b>Placement</b> that selects cluster <b>labels</b>; the hub matches placements against the
arriving through zero-touch provisioning, and nobody applies apps to clusters by hand. Each app carries a
Placement that selects cluster labels; the hub matches placements against the
labels a cluster wears, and the chosen cluster is given its assignment - push-style from the
hub by default, or in a pull model where each cluster runs its own reconciler. Labelling a
cluster changes its placement eligibility; the lifecycle and cleanup of what lands still follow
@@ -481,11 +480,11 @@
<h3>Keep plaintext secret values out of Git</h3>
<p>In this architecture, plaintext secret values do not live in Git: Git stores the
ExternalSecret reference, and the vault stores the value. Commit a plaintext secret and you
should treat it as <b>compromised from that moment</b> - deleting it later does not guarantee
should treat it as compromised from that moment - deleting it later does not guarantee
it is gone from history, forks, clones, CI caches or backups. So Git carries an ExternalSecret
naming a <b>logical key</b>; a vault (Azure Key Vault in the worked example) holds the value;
naming a logical key; a vault (Azure Key Vault in the worked example) holds the value;
an operator inside the cluster exchanges one for the other at runtime. Rotation happens in the
vault, not as a commit - which is why the vault sits apart from the pipeline in the scene
vault, not as a commit, which is why the vault sits apart from the pipeline in the scene
above.</p>
<p class="sayit">Git holds the shape of the secret. The vault holds the secret.</p>
<p class="closeline">Delivery becomes a property rather than an event: the cluster converges on
@@ -506,11 +505,11 @@
<header class="chead">
<p class="eyebrow">Course VII &middot; The image supply chain</p>
<h2 id="c7h" class="thesis">A tag is a promise.<br><em>A digest is a fact.</em></h2>
<p class="lede"><span class="mono">myapp:latest</span> feels like a name. It is a sticky note - a
<p class="lede"><span class="mono">myapp:latest</span> feels like a name. It is a sticky note: a
mutable reference anyone with push rights can move from one image to another. Registries may
audit tag updates, but the tag itself carries no immutability guarantee. The digest - the
sha256 of the image <i>manifest</i>, the small JSON index listing an image's layers, and no
relation to the YAML manifests you apply to a cluster - identifies the exact uploaded content.
audit tag updates, but the tag itself carries no immutability guarantee. The digest is the
sha256 of the image <i>manifest</i> (the small JSON index listing an image's layers, unrelated
to the YAML manifests you apply to a cluster). It identifies the exact uploaded content.
A local image ID printed by a build is not necessarily the same value, so pin the registry
digest that consumers actually pull.</p>
</header>
@@ -536,27 +535,27 @@
different code from one manifest - the second node pulled an hour later, after a re-push.
Nobody changed the YAML.</p>
<h3>Why a fleet pulls once</h3>
<p>Between the build and the node sits the registry chain. Upstream, a managed registry - Azure
Container Registry in the worked example - holds what CI built. In front of the cluster sits a
<b>mirror</b>: a pull-through cache like zot. On a cache miss the mirror fetches the artefact
<p>Between the build and the node sits the registry chain. Upstream, a managed registry (Azure
Container Registry in the worked example) holds what CI built. In front of the cluster sits a
mirror: a pull-through cache like zot. On a cache miss the mirror fetches the artefact
from upstream; subsequent requests can be served locally while that content stays cached. Rate
limits, egress cost, disconnected sites,
control - one place to gate and audit what enters. OpenShift formalises the re-route with
image mirror rules, and carries a sharp edge: digest-mirror rules rewrite <b>digest pulls
only</b>, so a by-tag pull silently skips them - unless you also add an
<b>ImageTagMirrorSet</b>, which is the rule type built for tag pulls. Pinning by digest is
image mirror rules, and carries a sharp edge: digest-mirror rules rewrite digest pulls
only, so a by-tag pull silently skips them - unless you also add an
ImageTagMirrorSet, which is the rule type built for tag pulls. Pinning by digest is
still the habit that makes the digest rules catch everything.</p>
<p>At real fleet scale the mirror itself tiers: a <b>central mirror in the cloud</b> fronts
<p>At real fleet scale the mirror itself tiers: a central mirror in the cloud fronts
upstream, and each site's mirror pulls from the centre rather than from upstream directly. A
new image ripples outward in layers - upstream to the centre, centre to each site as it asks,
site to its nodes over the LAN - rather than every site hitting upstream at the same moment.
site to its nodes over the LAN, rather than every site hitting upstream at the same moment.
With a site configured to consume only its local mirror, its node image traffic stays on the
local network.</p>
<p class="sayit">Tiered mirrors fan the load out in layers rather than all at once.</p>
<h3>Build once, promote by copy</h3>
<p>Every rebuild is a <i>new</i> artefact - in practice a different digest (reproducible builds
are the deliberate exception), untested by the stages before it.
So build <b>once</b>, then promote the same digest through environments by copying, registry to
So build once, then promote the same digest through environments by copying, registry to
registry - dev proves the exact bytes prod will run. For a multi-architecture image,
<span class="mono">skopeo copy --all</span> copies the complete image list; add
<span class="mono">--preserve-digests</span> when promotion requires the destination to keep
@@ -600,15 +599,15 @@
<h3>You are writing a program, not YAML</h3>
<p>Helm templates are Go text/template: <span class="mono">{{ .Values.device.address }}</span>
is a pipeline walking a values object. <span class="mono">_helpers.tpl</span> commonly holds
<b>named templates</b> - reusable partials that manifests can <span class="mono">include</span>
- rather than ordinary functions invoked directly. You are not writing YAML; you are writing a
named templates - reusable partials that manifests can <span class="mono">include</span>
, rather than ordinary functions invoked directly. You are not writing YAML; you are writing a
program whose output is YAML, so render locally, read the output, and lint what came out rather
than what went in.</p>
<p class="sayit">Review the render, not just the template.</p>
<p class="fieldnote"><b>Field note.</b> Argo CD renders charts with
<span class="mono">helm template</span> rather than running <span class="mono">helm
install</span>, so the lifecycle differs from Helm's own. <span class="mono">lookup</span>
comes back empty - there is no live cluster at render time. Hooks are not dead, though: Argo
comes back empty, because there is no live cluster at render time. Hooks are not dead, though: Argo
maps Helm hooks onto its sync phases (pre-install and pre-upgrade become PreSync, post-install
and post-upgrade become PostSync), while a few - rollback and test hooks - have no equivalent
at all. Render the way your deployer renders, and check where your hooks actually land.</p>
@@ -623,7 +622,7 @@
<h3>Make the template refuse</h3>
<p>A template that renders whatever it is given just moves the failure downstream. A production-grade
chart carries a <span class="mono">values.schema.json</span>: a context missing a required
value fails at render time, in the pipeline, with a message naming the field - not months
value fails at render time, in the pipeline, with a message naming the field, not months
later as enforcement pointed at nothing.</p>
<p class="fieldnote"><b>Field note.</b> The failure you want is the render that refuses. It
costs a red pipeline. The alternative reports healthy the whole time.</p>
@@ -644,7 +643,7 @@
<p class="eyebrow">Appendix &middot; The dependency ledger</p>
<h2 id="apxh" class="thesis">Every toolchain stands on<br><em>services it does not run.</em></h2>
<p class="lede">The arc reads like a closed machine: repo to reconciler to registry to node. It
is not closed. Three load-bearing pieces live outside the cluster - and the honest move is to
is not closed. Three load-bearing pieces live outside the cluster, and the honest move is to
write down what leans on them, and what happens when they are down.</p>
</header>
<figure class="still">