From e769007e0d10f541290f6d88763120bc83ca1202 Mon Sep 17 00:00:00 2001 From: jwright Date: Mon, 17 Aug 2026 18:49:03 +1000 Subject: [PATCH] technical accuracy pass: kubelet/runtime split, image vs container layer, exit 137 = SIGKILL, ArgoCD maps helm hooks, selfHeal/prune opt-in, scoped absolutes (reviewer findings) --- content/course-VI-gitops.md | 14 ++-- content/course-VIII-helm-contexts.md | 9 +-- pilot/arc.tpl.html | 96 +++++++++++++++++----------- pilot/manifest-docker.json | 4 +- pilot/manifest-node.json | 4 +- pilot/manifest-pod.json | 2 +- 6 files changed, 78 insertions(+), 51 deletions(-) diff --git a/content/course-VI-gitops.md b/content/course-VI-gitops.md index 01d1037..0f07cfe 100644 --- a/content/course-VI-gitops.md +++ b/content/course-VI-gitops.md @@ -21,8 +21,9 @@ Course II taught the reconciliation loop: desired vs actual, fix the difference, ArgoCD is the same habit applied to DELIVERY. An Application names a repo + path + revision (the GitHub linkage: watch this branch of this repository); the controller renders what it finds there, compares it against the live cluster, and shows the diff. Sync = make actual -match desired. The deploy button is a git commit; the audit log is git log; code review IS -change control. +match desired. The deploy button is a git commit; the change history is git log; code review IS +change control. (Accuracy: git records INTENT - the cluster's own audit log + the reconciler's +sync history still hold what actually happened.) Say it like you mean it: **kubectl apply is for archaeology, not for change.** ## Move 2 - Pull, not push - the security inversion @@ -31,14 +32,17 @@ GitHub Action holds a credential that can touch the cluster. The agent inside th holds a read-only deploy key to the repo - the trust arrow points OUT. Compromise the build system and you can propose a change (a commit, which is visible); you cannot reach into production. Drift works the same way: hand-edit a live object and the controller flags it -OutOfSync - with selfHeal on, it puts it back; with prune on, what leaves git leaves the -cluster. Rollback is `git revert`, which is why commit hygiene is an operational skill. +OutOfSync - with selfHeal ENABLED it puts it back; with prune ENABLED what leaves git leaves +the cluster. Both are opt-in (2026-08-17 review): without them the reconciler reports drift and +waits for a human. Scope the pull-model claims as 'in this architecture' - push-based delivery +is still GitOps to many. Rollback is `git revert`, which is why commit hygiene is an operational skill. FIELD NOTE (amber): fighting the reconciler by SSH/kubectl is arm-wrestling a machine that does not get tired (Course II's note, now with a face). The fix is always upstream, in git. ## Move 3 - The one thing git never holds Git holds everything - manifests, charts, config, the whole desired state - except secrets. -A secret in git is published, forever, to everyone who ever clones. So the pattern splits +A plaintext secret committed to git is COMPROMISED from that moment - deleting it later does +not guarantee it is gone from history, forks, clones, CI caches or backups. So the pattern splits the reference from the value: git carries an ExternalSecret manifest naming a LOGICAL key ("this app needs db-password from the vault"); a vault - Azure Key Vault as the worked example, any ESO-supported store works - holds the value; an operator inside the cluster diff --git a/content/course-VIII-helm-contexts.md b/content/course-VIII-helm-contexts.md index 0c43705..60156d9 100644 --- a/content/course-VIII-helm-contexts.md +++ b/content/course-VIII-helm-contexts.md @@ -24,10 +24,11 @@ every manifest calls, and `include`/`toYaml`/`default` compose them. The mental you are not writing YAML, you are writing a program whose OUTPUT is YAML - so render locally (`helm template`), read the output, and lint what came out, not what went in. Say it like you mean it: **review the render, not just the template.** -FIELD NOTE (amber): ArgoCD deploys charts by running `helm template` - a pure render. -Anything that needs a live cluster or an install lifecycle (`lookup`, hooks) is silently -dead there. A chart that works under `helm install` can behave differently under GitOps - -render the way your deployer renders. +FIELD NOTE (amber, CORRECTED 2026-08-17): ArgoCD renders with `helm template` rather than +running `helm install`, so the lifecycle differs. `lookup` comes back empty (no live cluster at +render). Hooks are NOT dead: Argo maps Helm hooks onto its sync phases (pre-install/pre-upgrade +-> PreSync, post-install/post-upgrade -> PostSync); rollback and test hooks have no equivalent. +Render the way your deployer renders and check where hooks land. ## Move 2 - Contexts: the cluster's whole voice is one small file The fleet pattern: the chart owns EVERYTHING structural - resources, probes, security diff --git a/pilot/arc.tpl.html b/pilot/arc.tpl.html index 8354c00..8a6a40e 100644 --- a/pilot/arc.tpl.html +++ b/pilot/arc.tpl.html @@ -13,8 +13,8 @@
  • Course 00Aperitif - three commands, and what actually just happened
  • Course IPodman - an image is a stack of frozen diffs
  • Course IIKubernetes - a cluster is a promise, not a place
    • -
  • Course IIIThe node - where intent becomes a process
    • -
  • Course IIIThe pod - one IP, shared fate
    • +
  • Course IIIaThe node - where intent becomes a process
    • +
  • Course IIIbThe pod - one IP, shared fate
  • Course IVThe traffic - pods are cattle, Services are the brand
  • Course VOpenShift - Kubernetes with opinions, and a security guard
  • Course VIGitOps - nobody deploys anything; the cluster syncs itself
    • @@ -96,11 +96,14 @@

    A stack of diffs wearing a name

    -

    An image is not a copy of a machine. It is a stack of read-only layers, each recording only - what changed from the one beneath. The base OS is pulled once and shared by every image on - the host; your dependencies sit on it; your code - usually the smallest layer, always the - most volatile - sits above that; and the writable layer is created fresh per container and - thrown away when it dies. Nothing you write there survives.

    +

    An image is not a copy of a machine. It is a stack of read-only layers, each + recording only what changed from the one beneath. Layers are content-addressed, so an + identical layer is stored once and reused by every image that references it - which is why + the second pull of anything is quick. A base sits at the bottom, your dependencies on it, + your code - usually the smallest layer, always the most volatile - above that.

    +

    The top slab in the scene is the odd one out: the writable layer is not part of the + image. The runtime adds it when it creates the container, and discards it when + that container dies. Nothing written there ships, and nothing written there survives.

    Change a layer and every layer above it must be rebuilt.

    The order is a caching decision

    The builder caches layer by layer, and a cached layer survives only while everything @@ -168,7 +171,7 @@

    -

    Course III · The node

    +

    Course IIIa · The node

    Where intent
    becomes a process.

    Everything so far was decision. This is the machine where a pod stops being paperwork and starts being a process - and the chain assembles link by link as you scroll.

    @@ -185,18 +188,19 @@
    -

    One component starts containers. One.

    -

    Every node runs a kubelet - and in the entire system it is the only component that - actually starts your container. It speaks CRI to containerd or CRI-O, which pulls the - image (through the mirror of Course VII) and launches it. A CNI plugin hands the pod +

    The kubelet decides; the runtime does

    +

    Every node runs a kubelet - the agent that owns what should be running there. It does + not create containers itself. It speaks CRI to containerd or CRI-O, and that + runtime pulls the image (through the mirror of Course VII) and actually creates and starts + the container, handing the low-level work to runc or crun. A CNI plugin hands the pod a real IP, and kube-proxy - or an eBPF datapath replacing it - makes Service addresses route to real pods. The kubelet drives, the runtime executes, the CNI wires, the proxy routes.

    -

    The kubelet is the only component that actually starts your container.

    +

    The kubelet decides what should run. The runtime is what starts it.

    Field note. Node NotReady? Check kubelet-to-API-server connectivity before anything else - a node that cannot phone home is presumed lost.

    The control plane never touches your workload. It writes intent; the - kubelet reads it and makes it flesh. Authority and execution meet exactly here, nowhere + kubelet turns intent into instructions, and the runtime turns instructions into processes. Authority and execution meet exactly here, nowhere else.

    Pre-reads: C-II  Further: cluster architecture · @@ -207,7 +211,7 @@

    -

    Course III · The pod

    +

    Course IIIb · The pod

    One IP,
    shared fate.

    A pod is not a container - it is a jacket around one or more. Scroll, and the capsule opens like a clamshell: shells apart, contents on display.

    @@ -226,7 +230,8 @@

    The jacket, not the container

    Everything inside the jacket shares a network namespace: one IP, localhost between - friends, volumes mountable by all. initContainers run first, in order, to completion - + friends. Volumes are declared once on the pod, but each container mounts the ones it needs - + sharing storage is opt-in, not automatic. initContainers run first, in order, to completion - gatekeepers under everything. Sidecars ride along with their own containers and their own jobs: proxy, logs, reload.

    Three probes, three different jobs

    @@ -234,8 +239,11 @@ truly hung. Confusing them is how healthy pods get executed - a slow start killed by an impatient liveness probe looks exactly like a crash.

    containerPort is documentation - the app still has to bind the port itself.

    -

    Field note. Exit 137 with reason Error is the kubelet's probe - kill; OOMKilled is memory. Same exit code, different crimes - read the reason.

    +

    Field note. Exit 137 is 128 + 9: the process was SIGKILLed. + It does not say by whom. The kernel's OOM killer surfaces as reason OOMKilled; a failed + liveness probe shows up in the pod's events; eviction and node pressure look different again. + One exit code, several possible crimes - read the termination reason and the events, never + the number alone.

    The pod is the smallest schedulable unit - the jacket, never the container. Once that distinction lands, half of Kubernetes networking stops being mysterious.

    @@ -274,7 +282,11 @@

    A pod failing its readiness probe silently leaves the pool. No error, no event at the client - traffic simply stops arriving. That is the feature: broken instances remove themselves. It is also the first place to look when traffic "disappears".

    -

    Never talk to a pod; talk to a Service.

    +

    The rule has deliberate exceptions: you address pods directly when debugging a specific + instance, and headless Services exist precisely so StatefulSet members can be reached + individually by stable DNS name. For ordinary application traffic, though, the Service is + the only address worth knowing.

    +

    For application traffic: never talk to a pod; talk to a Service.

    Field note. "The network is broken" after a deploy is usually readiness telling the truth about your app - not the network lying about your packets.

    The dark pod in the scene is not an error state - it is the system @@ -332,8 +344,9 @@

    Course VI · GitOps

    Nobody deploys anything.
    The cluster syncs itself.

    The mental model everyone arrives with: someone with credentials pushes manifests - at the cluster. In GitOps nothing is pushed. A repository holds the desired state, an agent - inside the cluster watches it, and the cluster pulls its own future from git. oc is for + at the cluster. In the architecture this course teaches, nothing is pushed: a repository holds + the desired state, an agent inside the cluster watches it, and the cluster pulls its own future + from git. oc is for looking; git is for changing.

    @@ -353,14 +366,19 @@ ArgoCD is the same habit applied to delivery. An Application names a repo, a path and a revision - watch this branch of this repository - and the controller renders what it finds there, compares it against the live cluster, and syncs the difference. The deploy button is a - git commit; the audit log is git log; code review is change control.

    + git commit; the change history is git log; code review is change control. Git is the + record of intent - the cluster's own audit log and the reconciler's sync history still + record what actually happened, including everything git never sees.

    oc apply is for archaeology, not for change.

    Pull, not push - the security inversion

    -

    The cluster pulls. No CI system, no laptop, no build pipeline holds a credential that - can touch it - the agent inside holds a read-only deploy key, and the trust arrow points out. - Compromise the build system and you can propose a change, which is visible; you cannot reach - into production. Hand-edit a live object and the controller flags it OutOfSync - selfHeal puts - it back, and what leaves git leaves the cluster. Rollback is git revert, +

    Here the cluster pulls. Push-based delivery exists and is still GitOps to many - this + is the stronger variant, and worth choosing deliberately: no CI system, no laptop, no build + pipeline holds a credential that can touch the cluster, because the agent inside holds a + read-only deploy key and the trust arrow points out. Compromise the build system and you can + propose a change, which is visible; you cannot reach into production. Hand-edit a live object + and the controller flags it OutOfSync - with selfHeal enabled it puts the object back, + and with prune enabled what leaves git leaves the cluster. Both are opt-in: without + them the reconciler reports the drift and waits for a human. Rollback is git revert, which is why commit hygiene is an operational skill.

    Field note. Fighting the reconciler over SSH is arm-wrestling a machine that does not get tired - C-II's warning, now with a face. The fix is always @@ -375,8 +393,9 @@ entitlements, reconciled like everything else.

    Label the cluster; the app follows.

    The one thing git never holds

    -

    Git holds everything except secrets - a secret in git is published, forever, to everyone who - ever clones. So the pattern splits the reference from the value: git carries an ExternalSecret +

    Git holds everything except secrets. Commit a plaintext secret and you should treat it as + compromised from that moment: deleting it later does not guarantee it is gone from + history, forks, clones, CI caches or backups. So the pattern splits the reference from the value: git carries an ExternalSecret naming a logical key; a vault - Azure Key Vault in the worked example - holds the value; an operator inside the cluster exchanges one for the other at runtime. Rotation happens in the vault, never as a commit. That is why the vault floats apart in the scene above: it is never @@ -488,10 +507,13 @@ functions every manifest calls. You are not writing YAML - you are writing a program whose output is YAML. So render locally, read the output, and lint what came out, not what went in.

    Review the render, not just the template.

    -

    Field note. ArgoCD deploys charts by running - helm template - a pure render. Anything needing a live cluster or an - install lifecycle (lookup, hooks) is silently dead there. Render the - way your deployer renders.

    +

    Field note. ArgoCD renders charts with + helm template rather than running helm + install, so the lifecycle differs from Helm's own. lookup + comes back empty - there is no live cluster at render time. Hooks are not dead, though: Argo + maps Helm hooks onto its sync phases (pre-install and pre-upgrade become PreSync, post-install + and post-upgrade become PostSync), while a few - rollback and test hooks - have no equivalent + at all. Render the way your deployer renders, and check where your hooks actually land.

    Contexts: the cluster's whole voice is one small file

    The chart owns everything structural - resources, probes, security, policy. Each cluster owns one values file: names, addresses, sizes, flags. The context is deliberately values-only; the @@ -712,16 +734,16 @@ b{color:var(--ink)} .mono{font-family:var(--mono);font-size:.9em;color:var(--cya .legend .b{font-size:.76rem;line-height:1.5;color:var(--dim);margin:0} .viewport.solo{grid-template-columns:1fr;max-width:980px} -.crt{position:absolute;inset:22% 26%;display:flex;flex-direction:column;justify-content:center; - gap:.35rem;padding:clamp(10px,2.5vw,26px);border:1px solid var(--line);border-radius:10px; +.crt{position:absolute;inset:18% 22%;display:flex;flex-direction:column;justify-content:center; + gap:.3rem;padding:clamp(10px,2.5vw,26px);border:1px solid var(--line);border-radius:10px; background:rgba(5,8,16,0.88);box-shadow:0 0 40px rgba(63,186,245,0.07) inset} .crt .line{font-family:var(--mono);font-size:clamp(.55rem,1.35vw,.85rem);line-height:1.5; - color:var(--ink);margin:0;white-space:nowrap;overflow:hidden} + color:var(--ink);margin:0;white-space:nowrap;padding-bottom:.12em} .crt .line.ok{color:var(--dim)} .crt .line.q{color:var(--magenta);margin-top:.5rem} body.fx .crt .line{opacity:0;transform:translateY(4px);transition:opacity .3s ease,transform .3s ease} body.fx .crt .line.on{opacity:1;transform:none} -@media (max-width:700px){.crt{inset:16% 12%}} +@media (max-width:700px){.crt{inset:10% 8%;gap:.2rem}.crt .line{line-height:1.7}} .prose{max-width:66ch;margin:0 auto;padding:5vh 24px 9vh} diff --git a/pilot/manifest-docker.json b/pilot/manifest-docker.json index 47eb034..c214f76 100644 --- a/pilot/manifest-docker.json +++ b/pilot/manifest-docker.json @@ -6,7 +6,7 @@ { "name": "base", "title": "Base image", - "blurb": "The OS layer everything else is stacked on. Pull it once, share it across every image on the host.", + "blurb": "The foundation everything stacks on. Layers are content-addressed - an identical layer is stored once and reused by every image that references it.", "z": 0, "points": [ [ @@ -144,7 +144,7 @@ { "name": "writable", "title": "Writable layer", - "blurb": "Created fresh per container, thrown away when it dies. Anything written here is not in the image.", + "blurb": "NOT part of the image. The runtime adds it when it creates the container, and discards it when that container dies.", "z": 0, "points": [ [ diff --git a/pilot/manifest-node.json b/pilot/manifest-node.json index 2bb40e5..ea08325 100644 --- a/pilot/manifest-node.json +++ b/pilot/manifest-node.json @@ -5,7 +5,7 @@ { "name": "kubelet", "title": "kubelet", - "blurb": "The node agent - the only component in the whole system that actually starts your container.", + "blurb": "The node agent. It decides what should be running and tells the runtime over CRI - it does not create containers itself.", "z": 0, "points": [ [ @@ -147,7 +147,7 @@ { "name": "engine", "title": "containerd / CRI-O", - "blurb": "Speaks CRI: pulls images, launches containers. The engine the kubelet drives.", + "blurb": "containerd or CRI-O: pulls the image and actually creates and starts the container.", "z": 0, "points": [ [ diff --git a/pilot/manifest-pod.json b/pilot/manifest-pod.json index 62dcd44..17f2fab 100644 --- a/pilot/manifest-pod.json +++ b/pilot/manifest-pod.json @@ -495,7 +495,7 @@ { "name": "discs", "title": "Volumes", - "blurb": "Shared mounts - the only place anything survives the writable layer's death.", + "blurb": "Declared once on the pod - but each container mounts the ones it needs. Sharing storage is opt-in.", "z": 0, "points": [ [