blog: open the EDGE AI, AUTOMATION and OBSERVABILITY arcs
build-and-deploy / build (push) Failing after 10m35s
build-and-deploy / build (push) Failing after 10m35s
Three arc-opening posts, each with hero and diagram: - probation-for-models (EDGE AI 01) - one-value-many-enforcers (AUTOMATION 01) - six-days-of-silence (OBSERVABILITY 01)
This commit is contained in:
Binary file not shown.
|
After Width: | Height: | Size: 119 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 73 KiB |
@@ -0,0 +1,45 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="470" viewBox="0 0 1180 470" role="img" aria-label="One required value in the cluster's values file feeds four rendered artefacts: the application config, the network attachment definition, the egress policy and the secondary-interface policy. Because every consumer renders from the same field, the four copies cannot disagree - there is nothing to update twice. Underneath, the other half of the pattern: the schema marks the field required, so a cluster that has not supplied the value fails at template time, in the pipeline, instead of shipping enforcement pointed at nothing.">
|
||||
<rect width="1180" height="470" fill="#ffffff"></rect>
|
||||
|
||||
<!-- title -->
|
||||
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">One source, four renders</text>
|
||||
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">divergence becomes something the tooling can no longer express</text>
|
||||
|
||||
<!-- source -->
|
||||
<rect x="80" y="160" width="240" height="90" rx="14" fill="#0c8fce" fill-opacity="0.07" stroke="#0c8fce" stroke-width="3"></rect>
|
||||
<text x="200" y="196" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12.5" fill="#0c8fce" font-weight="700">cluster values</text>
|
||||
<text x="200" y="218" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12" fill="#0e1726">device address · one field</text>
|
||||
<text x="200" y="272" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">lives once · edited once</text>
|
||||
|
||||
<!-- fan lines -->
|
||||
<g stroke="#64748b" stroke-width="1.5" fill="none">
|
||||
<path d="M 320 205 C 420 205 460 105 560 105"></path>
|
||||
<path d="M 320 205 C 420 205 460 172 560 172"></path>
|
||||
<path d="M 320 205 C 420 205 460 239 560 239"></path>
|
||||
<path d="M 320 205 C 420 205 460 306 560 306"></path>
|
||||
</g>
|
||||
|
||||
<!-- consumers -->
|
||||
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12" fill="#0e1726">
|
||||
<rect x="560" y="82" width="300" height="46" rx="10" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="1.5"></rect>
|
||||
<text x="580" y="110">application config</text>
|
||||
<rect x="560" y="149" width="300" height="46" rx="10" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="1.5"></rect>
|
||||
<text x="580" y="177">network attachment</text>
|
||||
<rect x="560" y="216" width="300" height="46" rx="10" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="1.5"></rect>
|
||||
<text x="580" y="244">egress rule</text>
|
||||
<rect x="560" y="283" width="300" height="46" rx="10" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="1.5"></rect>
|
||||
<text x="580" y="311">secondary-interface policy</text>
|
||||
</g>
|
||||
<g font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">
|
||||
<text x="890" y="110">what the workload calls</text>
|
||||
<text x="890" y="177">the address it answers on</text>
|
||||
<text x="890" y="244">what the traffic may do</text>
|
||||
<text x="890" y="311">same, for the second leg</text>
|
||||
</g>
|
||||
<text x="710" y="352" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12" font-weight="700" fill="#7c3aed">never independent copies — they render, they don't remember</text>
|
||||
|
||||
<!-- required strip -->
|
||||
<rect x="80" y="378" width="1020" height="58" rx="12" fill="#c026d3" fill-opacity="0.05" stroke="#c026d3" stroke-width="1.5"></rect>
|
||||
<text x="100" y="402" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12.5" font-weight="700" fill="#c026d3">absence fails loudly</text>
|
||||
<text x="100" y="422" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11.5" fill="#0e1726">field required in the schema → a cluster without it fails at template time, in the pipeline — not months later as policy pointed at nothing</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.1 KiB |
@@ -0,0 +1,79 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="520" viewBox="0 0 1180 520" role="img" aria-label="The probation ladder for downloaded models, left to right. Arrive: the artefact is more than weights - tokeniser code, an executable chat template, a loader gated by trust_remote_code - pinned by digest with its runtime. Probation tier: a Sandbox resource wraps a pod whose kata runtime class boots a per-pod KVM microVM with its own guest kernel, inside a zero-peer default-deny namespace holding zero credentials; the broker tier above holds all real keys. Observe: real workloads through the harness build a record of egress and behaviour against the written declaration. Gate: when the record matches the declaration over time the model promotes to the standard serving tier; unproven models stay in probation indefinitely. A hardware boundary separates the container from the node kernel for the whole supervised shift.">
|
||||
<rect width="1180" height="520" fill="#ffffff"></rect>
|
||||
|
||||
<!-- title -->
|
||||
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">The probation ladder</text>
|
||||
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">a new model is a new hire — promotion is earned on the record, never granted on arrival</text>
|
||||
|
||||
<!-- spine -->
|
||||
<line x1="80" y1="105" x2="1100" y2="105" stroke="#64748b" stroke-width="1.5"></line>
|
||||
<circle cx="200" cy="105" r="5" fill="#7c3aed"></circle>
|
||||
<circle cx="530" cy="105" r="5" fill="#0c8fce"></circle>
|
||||
<circle cx="830" cy="105" r="5" fill="#64748b"></circle>
|
||||
<circle cx="1020" cy="105" r="5" fill="#c026d3"></circle>
|
||||
|
||||
<!-- ARRIVE -->
|
||||
<text x="200" y="140" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#7c3aed">ARRIVE</text>
|
||||
<rect x="80" y="152" width="240" height="230" rx="14" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="2"></rect>
|
||||
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">
|
||||
<text x="100" y="182">not just weights:</text>
|
||||
<text x="100" y="204">+ tokeniser code</text>
|
||||
<text x="100" y="224">+ executable chat template</text>
|
||||
<text x="100" y="244">+ loader (trust_remote_code)</text>
|
||||
<text x="100" y="264">+ sometimes its own runtime</text>
|
||||
</g>
|
||||
<text x="100" y="300" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12" font-weight="700" fill="#7c3aed">pinned by digest,</text>
|
||||
<text x="100" y="318" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12" font-weight="700" fill="#7c3aed">artefact + runtime</text>
|
||||
<text x="100" y="352" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">the CV: an account name and</text>
|
||||
<text x="100" y="368" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">a download count — uncheckable</text>
|
||||
|
||||
<!-- PROBATION (highlighted) -->
|
||||
<text x="530" y="140" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#0c8fce">PROBATION — the supervised shift</text>
|
||||
<rect x="360" y="152" width="340" height="230" rx="14" fill="#0c8fce" fill-opacity="0.07" stroke="#0c8fce" stroke-width="3"></rect>
|
||||
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">
|
||||
<text x="380" y="182">Sandbox CR → pod → runtimeClass kata</text>
|
||||
<text x="380" y="204">→ per-pod KVM microVM,</text>
|
||||
<text x="392" y="222">its OWN guest kernel</text>
|
||||
<text x="380" y="248">namespace: default-deny, zero peers</text>
|
||||
<text x="380" y="268">credentials mounted: none</text>
|
||||
<text x="380" y="288">egress: only what is declared</text>
|
||||
<text x="380" y="308">keys: held by the broker tier above</text>
|
||||
</g>
|
||||
<line x1="380" y1="326" x2="680" y2="326" stroke="#0c8fce" stroke-width="1.5" stroke-dasharray="6 5"></line>
|
||||
<text x="380" y="348" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" font-weight="700" fill="#0c8fce">hardware boundary: an escape lands in a guest kernel,</text>
|
||||
<text x="380" y="366" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" font-weight="700" fill="#0c8fce">not on the node</text>
|
||||
|
||||
<!-- OBSERVE -->
|
||||
<text x="830" y="140" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#64748b">OBSERVE</text>
|
||||
<rect x="740" y="152" width="180" height="230" rx="14" fill="#64748b" fill-opacity="0.04" stroke="#64748b" stroke-width="2"></rect>
|
||||
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">
|
||||
<text x="758" y="182">real workloads</text>
|
||||
<text x="758" y="202">through the</text>
|
||||
<text x="758" y="222">harness</text>
|
||||
<text x="758" y="252">the record:</text>
|
||||
<text x="758" y="272">egress log vs</text>
|
||||
<text x="758" y="292">declaration</text>
|
||||
<text x="758" y="312">behaviour vs</text>
|
||||
<text x="758" y="332">expectation</text>
|
||||
</g>
|
||||
<text x="758" y="366" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">long enough, boring enough</text>
|
||||
|
||||
<!-- GATE -->
|
||||
<text x="1020" y="140" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#c026d3">GATE</text>
|
||||
<rect x="950" y="152" width="150" height="230" rx="14" fill="#c026d3" fill-opacity="0.05" stroke="#c026d3" stroke-width="2"></rect>
|
||||
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11">
|
||||
<text x="968" y="188" fill="#0c8fce">record matches</text>
|
||||
<text x="968" y="208" fill="#0c8fce">→ PROMOTE to</text>
|
||||
<text x="980" y="228" fill="#0c8fce">serving tier</text>
|
||||
<text x="968" y="266" fill="#c026d3">unproven</text>
|
||||
<text x="968" y="286" fill="#c026d3">→ stays in the</text>
|
||||
<text x="980" y="306" fill="#c026d3">VM. forever</text>
|
||||
<text x="980" y="326" fill="#c026d3">if need be</text>
|
||||
</g>
|
||||
<text x="968" y="362" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">permanent probation</text>
|
||||
<text x="968" y="378" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">costs almost nothing</text>
|
||||
|
||||
<!-- footer -->
|
||||
<text x="590" y="452" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12.5" font-weight="700" fill="#0e1726">trust is a history, not a property — observed behaviour, under constraint, accumulated until it's boring</text>
|
||||
<text x="590" y="478" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#64748b">the tax (seconds to boot, 350 MiB per microVM, batch-shaped work) isn't a flaw in the ladder — the tax IS the ladder</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 7.3 KiB |
@@ -0,0 +1,44 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="500" viewBox="0 0 1180 500" role="img" aria-label="Two views of the same six days. What the platform saw: the container running, the process alive, the port answering, logs present - every default check passing. What was true: the tunnel the workload exists to use was down, nothing had moved for six days, and the retry backoff made the logs quieter each day. A timeline underneath runs from the failure on day zero to discovery on day six - found by a person asking a question, not by the stack. The fix strip: probe the capability, not the container - the tunnel's own health check gates the workload, so health only reports when traffic genuinely egresses.">
|
||||
<rect width="1180" height="500" fill="#ffffff"></rect>
|
||||
|
||||
<!-- title -->
|
||||
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">Running was never the job</text>
|
||||
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">the gap between "the process is running" and "the process is doing its job" is where invisible outages live</text>
|
||||
|
||||
<!-- platform view -->
|
||||
<text x="315" y="112" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#0c8fce">WHAT THE PLATFORM SAW</text>
|
||||
<rect x="70" y="126" width="490" height="170" rx="14" fill="#0c8fce" fill-opacity="0.05" stroke="#0c8fce" stroke-width="2"></rect>
|
||||
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12" fill="#0e1726">
|
||||
<text x="100" y="160">container: Running</text>
|
||||
<text x="100" y="184">process: alive, answers, holds its port</text>
|
||||
<text x="100" y="208">every default check: passing</text>
|
||||
<text x="100" y="232">logs: present — and getting quieter</text>
|
||||
</g>
|
||||
<text x="100" y="272" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">entirely satisfied · no reason to think otherwise</text>
|
||||
|
||||
<!-- truth view -->
|
||||
<text x="865" y="112" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#c026d3">WHAT WAS TRUE</text>
|
||||
<rect x="620" y="126" width="490" height="170" rx="14" fill="#c026d3" fill-opacity="0.04" stroke="#c026d3" stroke-width="2"></rect>
|
||||
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12" fill="#0e1726">
|
||||
<text x="650" y="160">the tunnel: down</text>
|
||||
<text x="650" y="184">traffic moved: none, for six days</text>
|
||||
<text x="650" y="208">retry backoff: growing — the failure</text>
|
||||
<text x="662" y="230">was quietly obscuring itself</text>
|
||||
</g>
|
||||
<text x="650" y="272" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">alive, responsive, and completely useless</text>
|
||||
|
||||
<!-- timeline -->
|
||||
<line x1="90" y1="340" x2="1090" y2="340" stroke="#64748b" stroke-width="1.5"></line>
|
||||
<circle cx="130" cy="340" r="6" fill="#c026d3"></circle>
|
||||
<circle cx="1050" cy="340" r="6" fill="#0c8fce"></circle>
|
||||
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11.5">
|
||||
<text x="130" y="322" text-anchor="middle" fill="#c026d3">day 0: tunnel dies</text>
|
||||
<text x="1050" y="322" text-anchor="middle" fill="#0c8fce">day 6: a person asks</text>
|
||||
</g>
|
||||
<text x="590" y="366" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">six days of nothing — discovered by accident, the way every unwatched failure is discovered</text>
|
||||
|
||||
<!-- fix strip -->
|
||||
<rect x="70" y="396" width="1040" height="70" rx="12" fill="#0c8fce" fill-opacity="0.05" stroke="#0c8fce" stroke-width="1.5"></rect>
|
||||
<text x="90" y="422" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12.5" font-weight="700" fill="#0c8fce">probe the capability, not the container</text>
|
||||
<text x="90" y="444" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11.5" fill="#0e1726">the tunnel's own health check gates the workload → passes only when traffic genuinely egresses → silent failure becomes a restart, then an alert</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.4 KiB |
@@ -0,0 +1,55 @@
|
||||
---
|
||||
title: "Consistency by construction beats consistency by discipline"
|
||||
date: 2026-08-17
|
||||
summary: "One address had to appear in the application config, the network attachment and two policy objects - four artefacts, four update paths. Rendering every one of them from a single required value made divergence something the tooling can no longer express."
|
||||
tags: ["automation", "helm", "iac", "openshift", "fleet"]
|
||||
draft: false
|
||||
hero: "/blog/one-value-many-enforcers.webp"
|
||||
heroAlt: "A single source node feeding four identical threads of light into four separate gates"
|
||||
---
|
||||
|
||||
Every cluster in a small fleet runs a workload that talks to one device across a second network
|
||||
interface, and that device's address has to appear in four artefacts. The application config, so the
|
||||
workload knows what to call. The network attachment, so it gets the right address on the right
|
||||
interface. The egress rule, so the traffic is permitted. And the policy governing the second
|
||||
interface, for the same reason.
|
||||
|
||||
Four places. Four chances for someone to update three of them.
|
||||
|
||||
<!-- DIAGRAM: one cluster value feeding the application config, the network attachment, the primary policy and the secondary policy - one source, four renders -->
|
||||
|
||||
|
||||
## The failure mode has no error message
|
||||
|
||||
That is not hypothetical - a version of it had already happened here. An address changed in one
|
||||
artefact and not another, and the mismatch produced no error. It produced a policy that permitted
|
||||
traffic to somewhere nothing lived, while the actual traffic went somewhere unpoliced. Everything
|
||||
reported healthy. The enforcement had simply stopped meaning anything.
|
||||
|
||||
Config that must agree across several artefacts will eventually disagree, because the update path
|
||||
requires a human to remember all of them at once, under time pressure, months after writing them.
|
||||
|
||||
## Template from one source
|
||||
|
||||
The change that held: the address lives once, in the cluster's values, and every artefact that
|
||||
needs it renders from there. The application config, the attachment definition, both policies - all
|
||||
templated from the same field.
|
||||
|
||||
Now they cannot disagree. Not "are unlikely to" - *cannot*, because there is nothing to update
|
||||
twice. Changing the address is one edit, and every enforcement point follows automatically because
|
||||
they were never independent copies to begin with.
|
||||
|
||||
## Make absence fail loudly
|
||||
|
||||
The other half is refusing to render without it. The schema marks the field required, so a cluster
|
||||
that has not supplied an address fails at template time with a message that names the missing field.
|
||||
That failure is a gift: it lands in the pipeline, before anything ships, instead of surfacing months
|
||||
later as enforcement pointed at nothing.
|
||||
|
||||
## The render that refuses
|
||||
|
||||
Discipline was the old control here, and it had already failed once. Construction does not tire and
|
||||
cannot half-finish an edit: every consumer renders from the one field, or nothing renders at all.
|
||||
The failure you want is the render that refuses, not the deployment that polices the wrong address.
|
||||
|
||||
*Since fixed: the address field is required in the schema, and a render without it fails the pipeline.*
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
title: "A new model is a new hire, not a new file"
|
||||
date: 2026-08-17
|
||||
summary: "A model pulled from the hub is not just weights - it ships tokeniser code, an executable chat template, loaders gated by trust_remote_code, sometimes its own runtime, and the ecosystem's default is to run all of it beside your credentials. So every new model starts on probation: a hardware-isolated microVM with its own guest kernel, a default-deny network, zero credentials - and promotion to the standard serving tier only when the observed record earns it."
|
||||
tags: ["ai", "security", "kata", "isolation", "homelab"]
|
||||
draft: false
|
||||
hero: "/blog/probation-for-models.webp"
|
||||
heroAlt: "A new arrival's first supervised shift: an android-like figure of light works alone inside a glass observation room on a vast dark facility floor, instruments watching from outside, the busy permanent floor glowing far beyond"
|
||||
---
|
||||
|
||||
Pull a model from the hub and notice what your own head does with it. A directory of large binary files arrives, so it gets filed under *data*. Weights. Tensors. Numbers - and numbers can't run.
|
||||
|
||||
Except that isn't what you downloaded. A model artefact ships tokeniser code. It ships a chat template - an executable template the runtime evaluates on every request. It often ships a loader gated by a flag whose name confesses everything, `trust_remote_code=True`, and sometimes it effectively ships its own runtime, pulled as a container image from someone else's registry. The ecosystem's default is to run all of this with cluster-level convenience: the same kernel as everything else you host, the same namespace, a service-account token mounted at the usual path. Treat a fresh model as data and you will, sooner or later, execute a stranger's Python next to your credentials.
|
||||
|
||||
I've argued before that untrusted code belongs in a VM, not a namespace; the downloaded model is that argument's most routinely ignored case. So the operating rule in this fleet is a hiring rule, not a file-handling one: **new models start on probation**. You didn't add an asset to storage. You hired a stranger off the internet - and a new hire doesn't get keys to anything on day one.
|
||||
|
||||
<!-- DIAGRAM: the probation ladder, left to right. ARRIVE: a model artefact (weights + tokeniser code + chat template + loader), pinned by digest with its runtime. PROBATION TIER (live machinery): Sandbox CR -> pod with runtimeClassName kata -> per-pod KVM microVM with its OWN guest kernel, inside a zero-peer default-deny namespace; zero credentials mounted; egress opened only to declared needs; broker tier above holds all real keys. OBSERVE: real workloads through the harness; the record = egress log + behaviour vs declaration. GATE: record matches declaration over time -> PROMOTE to the standard serving tier behind the gateway; unproven -> stays in probation indefinitely (cheap). Highlight the hardware boundary between the container and the node kernel. -->
|
||||
|
||||
|
||||
## Admit what you actually downloaded
|
||||
|
||||
Consider what you'd normally know about a new starter: references you can call, a work history someone vouches for, an interview where you watched them think. Now inventory what you know about a model pulled from a public hub: an account name, a download count, a README written by the account. That's the whole CV, and none of it is checkable.
|
||||
|
||||
Meanwhile the artefact's executable surface is wider than most people's mental picture of it. The older checkpoint formats are pickles, and unpickling is code execution - safetensors exists precisely because "loading the weights" used to mean "running a program". Custom architectures ask for `trust_remote_code`, which imports and runs whatever Python the repository carries, in your process, with your permissions. Even the innocuous chat template is a small program, executed on every request. Little of this is malice; nearly all of it is engineering convenience. But convenience is how a stranger's code ends up running beside a token that can list every secret in the namespace.
|
||||
|
||||
You cannot call this candidate's references. The only reference check available is watching it work - which is what a probation period is.
|
||||
|
||||
## Give the stranger a supervised shift
|
||||
|
||||
The probation tier here is not a policy document; it's a namespace, and everything in it is live and verified. Kata Containers is baked into the node's install image, and a pod that sets `runtimeClassName: kata` doesn't get a slice of the host kernel - it boots inside its own KVM microVM, with its own guest kernel. Not a claim from the docs: run `uname -r` inside such a pod and it reads 6.18.28 while the node underneath runs 6.18.34. A workload that escapes its container has escaped into a guest VM, with a hardware boundary still between it and the node.
|
||||
|
||||
A `Sandbox` resource (kubernetes-sigs/agent-sandbox) wraps that pod and owns its lifecycle, and the namespace floor does the supervising. The network is default-deny with zero peers - a fresh sandbox can pull its image and reach nothing else until someone writes down, explicitly, the handful of addresses it's allowed. Admission rejects any pod that tries to start in that namespace without the microVM runtime, so nobody can absent-mindedly schedule a shortcut. And the tier holds no credentials at all, by design: anything a workload legitimately needs is asked for through the broker layer above it, which holds the real keys - the pattern from [an agent should never hold the key it's using](/blog/broker-pattern).
|
||||
|
||||
That's a supervised shift in full. Escorted everywhere it goes, nothing in its pockets, every outside contact through a chaperone - and the building stays standing even if the new starter turns out to be hostile.
|
||||
|
||||
## Promote on the record, not the calendar
|
||||
|
||||
Here is the honest line between proven and doctrine. The machinery above is live: the microVM boots, the guest kernel differs, the deny floor holds - all verified. The ladder is the operating rule that machinery was built to serve, and it goes like this.
|
||||
|
||||
A candidate model runs its evaluation harness inside the microVM, pinned by digest - the exact artefact and the exact runtime, not a tag that can drift under you. Its declared needs are written down first: these endpoints, this storage, nothing else. Then the record accumulates over real workloads. Egress matches the declaration, or it doesn't. Behaviour matches expectation, or it doesn't - no surprise network, no surprise syscalls, no creative interpretation of its job. When the record is long enough and boring enough, probation ends: the model is made permanent and moves to the standard serving tier behind the gateway, where the fast path lives.
|
||||
|
||||
The quiet half of the rule is the better half: anything unproven simply stays in the VM. Forever, if need be. Permanent probation costs almost nothing - the model still works, still answers, still earns its keep on supervised shifts. Promotion is a decision you can decline to make indefinitely, and the candidate has no grounds to complain.
|
||||
|
||||
## Pay the probation tax without flinching
|
||||
|
||||
None of this is free. A microVM boots in seconds, not milliseconds. I/O crosses a virtualisation boundary and pays for the trip. Every pod carries its own guest kernel in memory - this fleet books 350 MiB of overhead against each microVM so the scheduler doesn't lie to itself about what fits. Probation-tier work is batch-shaped, not interactive.
|
||||
|
||||
That's fine, because it's what probation means. You don't measure a supervised shift by throughput; you measure it by what it reveals. The standard tier - shared kernel, warm caches, low latency - exists because the promotion is earned, and a reward only means something if the default is slower. The tax isn't a flaw in the ladder. The tax *is* the ladder.
|
||||
|
||||
## Trust is a history, not a property
|
||||
|
||||
Trust is not a property of software. No scanner emits it, no licence contains it, and a clean hash only proves that today's stranger is the same stranger as yesterday - it says nothing about what the stranger does. Trust is a history: observed behaviour, under constraint, accumulated until it's boring. And it's graduated, never granted on arrival.
|
||||
|
||||
The reason to bother is worth saying plainly. A model is not a config value you swap on a hunch; it is the part of the system that decides what is true, and it arrives with a wider executable surface than almost anyone pictures. Weights, agents, plugins, anything that turns up executable - the ladder is the same. Start it where an escape lands in a guest kernel, where the network answers to a written declaration, where there are no credentials to spend. Let the record grow. Promote on the record, or not at all.
|
||||
|
||||
Give every new arrival a place to build a history where the worst it can do is bounded.
|
||||
|
||||
*Live on the fleet's untrusted tier: Kata 3.31.0 baked into a Talos node image, RuntimeClass `kata` booting per-pod KVM microVMs (guest kernel 6.18.28 on a 6.18.34 host), agent-sandbox v0.4.6 wrapping them in Sandbox resources, a zero-peer default-deny namespace holding not one credential - the supervised shift the next downloaded model walks into.*
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
title: "Six days of nothing, and nothing noticed"
|
||||
date: 2026-08-17
|
||||
summary: "A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied - because nothing was watching the thing that had actually failed."
|
||||
tags: ["observability", "monitoring", "reliability", "alerting", "homelab"]
|
||||
draft: false
|
||||
hero: "/blog/six-days-of-silence.webp"
|
||||
heroAlt: "A wall of uniformly lit all-clear indicator lamps above a floor pipeline that has visibly run dry"
|
||||
---
|
||||
|
||||
It was found by accident. Someone asked how the downloads were going, and the answer was that they
|
||||
had not gone anywhere in six days. The tunnel that carries them had failed and the process
|
||||
responsible had kept running - alive, responsive, and completely useless.
|
||||
|
||||
Nothing alerted, because nothing was watching the tunnel. The container was up. The platform had no
|
||||
reason to think otherwise.
|
||||
|
||||
<!-- DIAGRAM: a container running with a healthy process while its tunnel is down, no probe covering the tunnel, and the six-day gap between failure and discovery -->
|
||||
|
||||
|
||||
## Running was never the job
|
||||
|
||||
The gap between "the process is running" and "the process is doing its job" is where invisible
|
||||
outages live. A tunnel client with a dead tunnel is still a healthy process. It answers, it holds
|
||||
its port, it logs. It simply is not carrying anything, and no default check in the stack has an
|
||||
opinion about that.
|
||||
|
||||
Worse, the failure was self-obscuring. Its internal retry backoff kept growing, so the logs got
|
||||
quieter over time rather than louder. By day six, the system was producing almost no evidence that
|
||||
anything was wrong.
|
||||
|
||||
## Health lives in the flow
|
||||
|
||||
What closed the gap was probing the capability rather than the container: the tunnel's own health
|
||||
check, which only passes when traffic genuinely egresses through it. That converts a silent failure
|
||||
into a restart and, eventually, an alert - a thing the platform can see and act on.
|
||||
|
||||
The general rule: for anything whose value is a *flow*, health means the flow works. Ask whether
|
||||
data recently moved, not whether the mover is alive.
|
||||
|
||||
## The question that found it
|
||||
|
||||
Health checks should assert the outcome the component exists to produce. A running process proves
|
||||
almost nothing about a system whose job is to move something, and a failure nobody is watching for
|
||||
will always be discovered the way this one was - by a person, on the day they happen to ask.
|
||||
|
||||
*Live in the lab: the tunnel's health check gates the container now - nothing reports healthy
|
||||
unless traffic egresses through it.*
|
||||
@@ -37,4 +37,13 @@ export const series: Record<string, { name: string; number: number }> = {
|
||||
"sleep-half-the-fleet": { name: "PLATFORM SERIES", number: 5 },
|
||||
"restore-or-rumour": { name: "PLATFORM SERIES", number: 6 },
|
||||
"stage-the-locks": { name: "PLATFORM SERIES", number: 7 },
|
||||
|
||||
// EDGE AI SERIES - models, trust and promotion at the edge
|
||||
"probation-for-models": { name: "EDGE AI SERIES", number: 1 },
|
||||
|
||||
// AUTOMATION SERIES - config as contract, interfaces built to be automated
|
||||
"one-value-many-enforcers": { name: "AUTOMATION SERIES", number: 1 },
|
||||
|
||||
// OBSERVABILITY SERIES - watching the thing that actually fails
|
||||
"six-days-of-silence": { name: "OBSERVABILITY SERIES", number: 1 },
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user