Purpose of this document

The Exploded Cluster teaches how modern container platforms work by taking them apart - literally. Each course is one machine drawn as a single exploded illustration, sliced into its real components and wired to your scroll, so the architecture moves while the words explain it. Start with the aperitif's three terminal commands; finish knowing how a change travels from a git commit to a running, secret-fed, digest-pinned workload on a fleet. Every claim comes from running these systems for real, and the library at the end links only to official documentation.

Course 00 · Aperitif

## Three commands.
What actually just happened?

The whole ceremony of shipping software fits in four lines of terminal. They work on your first day and stay mysterious for years. Scroll - the shell comes off first.

$ podman build -t shop/api:1.4.2 .

=> exported image sha256:9f2c41...

$ podman push shop/api:1.4.2

$ oc apply -f deploy.yaml

deployment.apps/api created

$ oc get pods

api-7d4b9c6-x2k4f 1/1 Running

...but what actually just happened?

An image got built - of what, exactly? Pushed - to where, and what travelled? Applied - which is not the same as launched. Running - according to whom? Every course below takes one of those words apart. The armour is already loose.

Course I · Podman

## An image is not a box.
It is a stack of frozen diffs.

Scroll, and the thing you keep calling "a container image" comes apart in your hands. Four layers. Each one only stores what changed from the layer under it - and here they rise one at a time, bottom up.

### A stack of diffs wearing a name

An image is not a copy of a machine. It is a stack of read-only layers, each recording only what changed from the one beneath. Layers are content-addressed, so an identical layer is stored once and reused by every image that references it, so a pull only fetches the layers you do not already have. A base sits at the bottom, your dependencies on it, your code - usually the smallest layer, always the most volatile - above that.

The top slab in the scene is the odd one out: the writable layer is not part of the image. The runtime adds it when it creates the container, and discards it when that container dies. Nothing written there ships, and nothing written there survives.

Change a layer and every layer above it must be rebuilt.

### The order is a caching decision

The builder caches layer by layer, and a cached layer survives only while everything beneath it is unchanged. Put COPY . . above your dependency install and you have told the builder to throw the dependency cache away every time one line of code changes. Dependencies first, code last - a Containerfile is a cache policy that happens to build software.

Field note. A rebuild that takes twenty minutes and one that takes twenty seconds are usually the same Containerfile with the lines swapped.

This stack explains the whole ecosystem above it: sharing explains why pulls are fast, immutability explains why a digest can name the exact bytes (Course VII), and the throwaway top layer explains why state needs volumes. One idea - frozen diffs - all the way down.

Pre-reads: none - start hereFurther: Podman get-started · Podman docs + builds

Course II · Kubernetes

## A cluster is a promise,
not a place.

You never tell Kubernetes how to run your app. You describe what you want - declarative intent - and the cluster works, forever, to make it true. Scroll, and the formation splits: the half that decides rises, the halves that run spread below.

### Desired against actual, forever

The habit underneath everything: the reconciliation loop - compare desired state against actual state, fix the difference, repeat. That oc apply didn't launch anything; it filed paperwork. The machine took it from there, and it never stops taking it from there: kill a pod and it returns, not because something noticed the crash but because the loop noticed the difference.

Kubernetes doesn't run your app - it reconciles it.

### One half decides, one half runs

The split in the scene is the split that makes everything else possible: a control plane that decides - holds the truth, schedules, reconciles - and worker nodes that run pods. The workers are deliberately interchangeable: identical, replaceable, cattle from day one. Authority does not live where the work happens.

Field note. If you are SSHing into nodes to "fix" things, you are arm-wrestling the reconciler - and it does not get tired. Change the desired state instead.

Everything the delivery arc teaches from Course VI onward is this loop wearing bigger clothes - git as the desired state, whole fleets as reconciled objects. Learn the promise once; it repeats all the way up.

Pre-reads: C-IFurther: Kubernetes overview · cluster architecture

Course IIIa · The node

## Where intent
becomes a process.

Everything so far was decision. This is the machine where a pod stops being paperwork and starts being a process - and the chain assembles link by link as you scroll.

### The kubelet decides; the runtime does

Every node runs a kubelet - the agent that owns what should be running there. It does not create containers itself. It speaks CRI to containerd or CRI-O, and that runtime pulls the image (through the mirror of Course VII) and actually creates and starts the container, handing the low-level work to runc or crun. A CNI plugin hands the pod a real IP, and kube-proxy - or an eBPF datapath replacing it - makes Service addresses route to real pods. The kubelet drives and the runtime executes - and it is the runtime, not the kubelet, that calls the CNI plugin to wire the pod up.

The kubelet decides what should run. The runtime is what starts it.

Field note. Node NotReady? Check kubelet-to-API-server connectivity before anything else - a node that cannot phone home is presumed lost.

The control plane never touches your workload. It writes intent; the kubelet turns intent into instructions, and the runtime turns instructions into processes. Authority and execution meet exactly here, nowhere else.

Pre-reads: C-IIFurther: cluster architecture · node components

Course IIIb · The pod

## One IP,
shared fate.

A pod is not a container - it is a jacket around one or more. Scroll, and the capsule opens like a clamshell: shells apart, contents on display.

### The jacket, not the container

Everything inside the jacket shares a network namespace: one IP, localhost between friends. Volumes are declared once on the pod, but each container mounts the ones it needs - sharing storage is opt-in, not automatic. initContainers run first, in order, to completion - gatekeepers under everything. Sidecars ride along with their own containers and their own jobs: proxy, logs, reload.

### Three probes, three different jobs

startup owns warm-up, readiness gates traffic, liveness restarts the truly hung. Confusing them is how healthy pods get executed - a slow start killed by an impatient liveness probe looks exactly like a crash.

containerPort is documentation - unless a Service targets it by name, or you use hostPort. Either way the app still has to bind the port itself.

Field note. Exit 137 is 128 + 9: the process was SIGKILLed. It does not say by whom. The kernel's OOM killer surfaces as reason OOMKilled; a failed liveness probe shows up in the pod's events; eviction and node pressure look different again. One exit code, several possible crimes - read the termination reason and the events, never the number alone.

The pod is the smallest schedulable unit - the jacket, never the container. Once that distinction lands, half of Kubernetes networking stops being mysterious.

Pre-reads: C-III's node scene aboveFurther: pods · the three probes

Course IV · The traffic

## Pods are cattle.
Services are the brand.

Pods die, respawn and change addresses - and traffic still arrives. Scroll, and the delivery route assembles checkpoint by checkpoint; watch what happens to the pod that stops answering.

### The stable name in front of the churn

A Service is the fixed point: a ClusterIP inside, a LoadBalancer at the edge, Ingress or the Gateway API doing host- and path-routing above. Clients hold the name; the pods behind it come and go without anyone being told.

### Readiness decides membership

A pod failing its readiness probe silently leaves the pool. No error, no event at the client - traffic simply stops arriving. That is the feature: broken instances remove themselves. It is also the first place to look when traffic "disappears".

The rule has deliberate exceptions: you address pods directly when debugging a specific instance, and headless Services exist precisely so StatefulSet members can be reached individually by stable DNS name. For ordinary application traffic, though, the Service is the only address worth knowing.

For application traffic: never talk to a pod; talk to a Service.

Field note. "The network is broken" after a deploy is usually readiness telling the truth about your app - not the network lying about your packets.

The dark pod in the scene is not an error state - it is the system working. Membership is earned per probe, continuously, forever.

Pre-reads: C-III's pod scene (readiness lives there)Further: Services and networking

Course V · OpenShift

## Kubernetes with opinions -
and a security guard.

OpenShift is a distribution of Kubernetes: same engine, opinionated chassis. Scroll, and the opinions bloom outward from the core they orbit.

### The doorman interviews every pod

The SCC is admission deciding what a pod may BE. The default, restricted-v2, runs your container as a random non-root UID - your image has to cope. Workloads that genuinely need privilege get a dedicated ServiceAccount bound to a minimal custom SCC, never the stock one.

### The platform runs itself

Routes predate Ingress and still rule here. Operators run machinery on your behalf - though the split matters: the platform's own operators are driven by the Cluster Version Operator, while OLM installs and upgrades the add-on Operators you choose from OperatorHub. The OS underneath is immutable - changed by MachineConfig, never by SSH. And SNO puts the entire promise on one box at the edge. At fleet scale the labels from Course VI decide which of these boxes runs what.

On OpenShift, admission is the interview - the SCC is the dress code.

Field note. Deployment stuck at 0/1 with no pod at all? The refusal happened above scheduling - read the ReplicaSet events. The error lives a level up.

Everything in the ring is opinion wrapped around the same core you already know. Learn the core once; the opinions are just house rules.

Pre-reads: C-IIFurther: OpenShift documentation · Red Hat OpenShift

Course VI · GitOps

## Nobody deploys anything.
The cluster syncs itself.

The mental model everyone arrives with: someone with credentials pushes manifests at the cluster. In the architecture this course teaches, nothing is pushed: a repository holds the desired state, an agent inside the cluster watches it, and the cluster pulls its own future from git. oc is for looking; git is for changing.

### The loop you already know, one level up

C-II taught the reconciliation loop: desired versus actual, fix the difference, repeat. ArgoCD is the same habit applied to delivery. An Application names a repo, a path and a revision - watch this branch of this repository - and the controller renders what it finds there, compares it against the live cluster, and syncs the difference. The deploy button is a git commit; the change history is git log; code review is change control. Git is the record of intent - the cluster's own audit log and the reconciler's sync history still record what actually happened, including everything git never sees.

oc is for archaeology. git is for change.

### Pull, not push - the security inversion

Here the cluster pulls. Push-based delivery exists and is still GitOps to many - this is the stronger variant, and worth choosing deliberately: no CI system, no laptop, no build pipeline holds a credential that can touch the cluster, because the agent inside holds a read-only deploy key and the trust arrow points out. Compromise the build system and you can propose a change, which is visible; you cannot reach into production through git. It can still push images - which is the other half of why a manifest should name the digest, not the tag. Hand-edit a live object and the controller flags it OutOfSync - with selfHeal enabled it puts the object back, and with prune enabled what leaves git leaves the cluster. Both are opt-in: without them the reconciler reports the drift and waits for a human. Rollback is git revert, which is why commit hygiene is an operational skill.

Field note. Fighting the reconciler over SSH is arm-wrestling a machine that does not get tired - C-II's warning, now with a face. The fix is always upstream, in git.

### At fleet scale, the label is the deploy button

Run many OpenShift clusters under a hub - RHACM, with edge clusters arriving through zero-touch provisioning - and nobody applies apps to clusters by hand. Each app carries a Placement that selects cluster labels; the hub matches placements against the labels a cluster wears, and the chosen cluster is given its assignment - push-style from the hub by default, or in a pull model where each cluster runs its own reconciler, which is the variant this arc teaches. Either way, labelling the cluster is the deploy action: attach the label and the app follows, remove it and the app leaves. The same pull model as above, one level bigger - a cluster's labels are its entitlements, reconciled like everything else.

Label the cluster; the app follows.

### The one thing git never holds

Git holds everything except secrets. Commit a plaintext secret and you should treat it as compromised from that moment: deleting it later does not guarantee it is gone from history, forks, clones, CI caches or backups. So the pattern splits the reference from the value: git carries an ExternalSecret naming a logical key; a vault - Azure Key Vault in the worked example - holds the value; an operator inside the cluster exchanges one for the other at runtime. Rotation happens in the vault, never as a commit. That is why the vault floats apart in the scene above: it is never absorbed into the pipeline.

Git holds the shape of the secret. The vault holds the secret.

Delivery stops being an event and becomes a property: the cluster is always converging on what the repository says. "Who deployed this?" becomes "who merged this?" - and that question always has an answer.

Pre-reads: C-II · Kubernetes concepts · git + pull requests Further: Argo CD · OpenShift GitOps · External Secrets · Azure Key Vault

Course VII · The image supply chain

## A tag is a promise.
A digest is a fact.

myapp:latest feels like a name. It is a sticky note - a mutable pointer anyone with push rights can peel off one image and press onto another, and nothing anywhere records that it moved. The digest - sha256 of the image manifest - is its actual name: same bytes, same digest, forever. (Careful: the sha256 a build prints is the local image ID, a different hash from the manifest digest the registry mints on push - the pushed one is what you pin.)

### Say the name properly

Three ways to name an image, in rising order of honesty: :latest (a moving target), :1.4.2 (a promise somebody keeps, until they re-push it), and name:1.4.2@sha256:... - a fact. The tag stays for human eyes; the digest does the pulling. Pin by digest and "what is running?" has exactly one answer.

Field note. :latest is how two nodes run different code from one manifest - the second node pulled an hour later, after a re-push. Nobody changed the YAML.

### Why a fleet pulls once

Between the build and the node sits the registry chain. Upstream, a managed registry - Azure Container Registry in the worked example - holds what CI built. In front of the cluster sits a mirror: a pull-through cache like zot. The fleet asks the mirror, the mirror asks upstream once, everything after is local. Rate limits, egress cost, disconnected sites, control - one place to gate and audit what enters. OpenShift formalises the re-route with image mirror rules, and carries a sharp edge: digest-mirror rules rewrite digest pulls only, so a by-tag pull silently skips them - unless you also add an ImageTagMirrorSet, which is the rule type built for tag pulls. Pinning by digest is still the habit that makes the digest rules catch everything.

At real fleet scale the mirror itself tiers: a central mirror in the cloud fronts upstream once, and every site's mirror pulls from the centre rather than from upstream directly. A new image ripples outward in layers - upstream to the centre, centre to each site as it asks, site to its nodes over the LAN - instead of every site hammering upstream at the same moment. Upstream sees one consumer; each site sees one hop; the nodes never leave the building.

Mirrors tier: the load fans out in layers, never all at once.

### Build once, promote by copy

Every rebuild is a new artefact - in practice a different digest (reproducible builds are the deliberate exception), untested by the stages before it. So build once, then promote the same digest through environments by copying, registry to registry - dev proves the exact bytes prod will run. On a multi-arch image that copy needs --all (and --preserve-digests to fail loudly rather than quietly), or you copy one architecture and the digest you promoted is not the digest that lands. Human tags ride along; the digest is the through-line.

If the digest changed, it is not a promotion - it is a new candidate.

Names that can move are convenient exactly until they move. Address content by what it is, and the supply chain stops being a chain of trust and becomes a chain of arithmetic.

Pre-reads: C-I · Kubernetes images Further: zot · Azure Container Registry · OpenShift image mirroring · skopeo

Course VIII · Helm

## A chart is a function.
The values file is the cluster speaking.

The default way to run one app on five clusters is five copies of the YAML, and the default result is five slightly different apps. The inversion: stop copying outputs and ship the function. A chart is a template with holes; each cluster supplies one small values file that fills them.

### It's Go under the braces

Helm templates are Go text/template: {{ .Values.device.address }} is a pipeline walking a values object, _helpers.tpl holds the named templates - the partials every manifest includes, not functions you can call bare. You are not writing YAML - you are writing a program whose output is YAML. So render locally, read the output, and lint what came out, not what went in.

Review the render, not just the template.

Field note. ArgoCD renders charts with helm template rather than running helm install, so the lifecycle differs from Helm's own. lookup comes back empty - there is no live cluster at render time. Hooks are not dead, though: Argo maps Helm hooks onto its sync phases (pre-install and pre-upgrade become PreSync, post-install and post-upgrade become PostSync), while a few - rollback and test hooks - have no equivalent at all. Render the way your deployer renders, and check where your hooks actually land.

### Contexts: the cluster's whole voice is one small file

The chart owns everything structural - resources, probes, security, policy. Each cluster owns one values file: names, addresses, sizes, flags. The context is deliberately values-only; the moment it carries its own manifests there are two owners for one object, and they will disagree. One value can feed many rendered artefacts - an address appearing in the app config, the network attachment and two policies renders from one field, so the copies cannot diverge - the lived version is on the blog.

The chart owns the shape. The context owns the numbers.

### Make the template refuse

A template that renders whatever it is given just moves the failure downstream. The grown-up chart carries a values.schema.json: a context missing a required value fails at render time, in the pipeline, with a message naming the field - not months later as enforcement pointed at nothing.

Field note. The failure you want is the render that refuses. It costs a red pipeline. The alternative reports healthy the whole time.

Fleet consistency is not a discipline problem, it is a construction problem. Divergence stops being something you police and becomes something the tooling cannot express.

Pre-reads: C-II · Kubernetes objects Further: Helm docs · chart template guide · Go text/template · Helm on OpenShift

Appendix · The dependency ledger

## Every toolchain stands on
services it does not run.

The arc reads like a closed machine: repo to reconciler to registry to node. It is not closed. Three load-bearing pieces live outside the cluster - and the honest move is to write down what leans on them, and what actually happens when they are down.

You have met these three before.

### GitHub

Where the desired state lives - the system of record the whole loop watches, through a read-only deploy key.

Leans on it: sync, rollback, change review, the "who merged this" answer.

When it is down: Kubernetes keeps running the last applied state indefinitely. The reconciler keeps self-healing only while its rendered manifests are still cached - hours, not forever, and gone after a restart. What stops is change. GitOps degrades to read-only, which is the graceful half of the design.

### Azure Key Vault

Where the secret values live - git carries the reference, the vault carries the value, an operator keeps them synced.

Leans on it: secret sync, rotation, the first deploy of anything that needs a credential.

When it is down: already-synced Secrets keep working - values are materialised in-cluster. What stops is rotation and new secrets. Survivable - unless you are inside a rotation window.

### zot

Where the fleet pulls from - a pull-through mirror between the cluster and the internet, and the control point for what enters. At fleet scale it tiers: one central mirror in the cloud fans out to per-site mirrors, layering the load.

Leans on it: every image pull on every node - boot, reschedule, scale-up, recovery.

When it is down: the sharpest edge. Upstream down + mirror up = nobody notices, provided the image is already cached - a cold entry still needs upstream. Mirror down on a mirror-only pull path = nothing new schedules unless the node already holds the image, and imagePullPolicy: Always turns a mirror outage into a hard stop.

None of these outages stop what is already running - they stop change, rotation and recovery, in that order of pain. Cache what you pull, split references from values, and let the cluster hold its last known truth without asking anyone's permission.

Further: GitHub docs · Azure Key Vault · zot · Kubernetes · Helm · Red Hat OpenShift

The whole arc · end to end

## One flow, no gaps.

Every course above is one stretch of the same journey. Here is the full run, drawn in the house blueprint style: the change lane, the shape lane, the artefact lane and the secret lane, all converging on one running workload.

The library

## Go to the sources.

Every technology this site teaches, one sentence each, official documentation only.

- Kubernetes The container orchestrator underneath everything here: declarative state, the reconciliation loop, pods, services and the machinery that keeps promises. kubernetes.io/docs
- Helm The package manager and Go-template engine that turns one chart plus a per-cluster values file into the rendered manifests a cluster actually runs. helm.sh/docs
- Red Hat OpenShift Kubernetes as an opinionated, supported distribution - SCCs, Routes and Operators, plus the fleet tooling (RHACM, zero-touch provisioning) that assigns apps to clusters by label. docs.redhat.com
- Argo CD The GitOps reconciler that watches a repository and keeps the cluster converged on what it finds there - the pull model that means nothing ever pushes to production. argo-cd.readthedocs.io
- GitHub Actions CI that builds and tests on every push - in this flow it makes artefacts and proposes changes, and never holds a credential that can touch a cluster. docs.github.com/actions
- Azure Key Vault The managed vault holding secret values, so git carries only references and rotation never has to become a commit. learn.microsoft.com/key-vault
- Azure Container Registry The managed upstream registry CI pushes to - where the digest is minted that everything downstream pins and promotes. learn.microsoft.com/container-registry