blog: open the EDGE AI, AUTOMATION and OBSERVABILITY arcs
build-and-deploy / build (push) Failing after 10m35s

Three arc-opening posts, each with hero and diagram:
- probation-for-models (EDGE AI 01)
- one-value-many-enforcers (AUTOMATION 01)
- six-days-of-silence (OBSERVABILITY 01)
This commit is contained in:
2026-08-17 01:50:25 +10:00
parent 6c0d7643a6
commit e90bf177a5
10 changed files with 338 additions and 0 deletions
Binary file not shown.

After

Width:  |  Height:  |  Size: 119 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

@@ -0,0 +1,45 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="470" viewBox="0 0 1180 470" role="img" aria-label="One required value in the cluster's values file feeds four rendered artefacts: the application config, the network attachment definition, the egress policy and the secondary-interface policy. Because every consumer renders from the same field, the four copies cannot disagree - there is nothing to update twice. Underneath, the other half of the pattern: the schema marks the field required, so a cluster that has not supplied the value fails at template time, in the pipeline, instead of shipping enforcement pointed at nothing.">
<rect width="1180" height="470" fill="#ffffff"></rect>
<!-- title -->
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">One source, four renders</text>
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">divergence becomes something the tooling can no longer express</text>
<!-- source -->
<rect x="80" y="160" width="240" height="90" rx="14" fill="#0c8fce" fill-opacity="0.07" stroke="#0c8fce" stroke-width="3"></rect>
<text x="200" y="196" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12.5" fill="#0c8fce" font-weight="700">cluster values</text>
<text x="200" y="218" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12" fill="#0e1726">device address &#183; one field</text>
<text x="200" y="272" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">lives once &#183; edited once</text>
<!-- fan lines -->
<g stroke="#64748b" stroke-width="1.5" fill="none">
<path d="M 320 205 C 420 205 460 105 560 105"></path>
<path d="M 320 205 C 420 205 460 172 560 172"></path>
<path d="M 320 205 C 420 205 460 239 560 239"></path>
<path d="M 320 205 C 420 205 460 306 560 306"></path>
</g>
<!-- consumers -->
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12" fill="#0e1726">
<rect x="560" y="82" width="300" height="46" rx="10" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="1.5"></rect>
<text x="580" y="110">application config</text>
<rect x="560" y="149" width="300" height="46" rx="10" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="1.5"></rect>
<text x="580" y="177">network attachment</text>
<rect x="560" y="216" width="300" height="46" rx="10" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="1.5"></rect>
<text x="580" y="244">egress rule</text>
<rect x="560" y="283" width="300" height="46" rx="10" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="1.5"></rect>
<text x="580" y="311">secondary-interface policy</text>
</g>
<g font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">
<text x="890" y="110">what the workload calls</text>
<text x="890" y="177">the address it answers on</text>
<text x="890" y="244">what the traffic may do</text>
<text x="890" y="311">same, for the second leg</text>
</g>
<text x="710" y="352" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12" font-weight="700" fill="#7c3aed">never independent copies &#8212; they render, they don't remember</text>
<!-- required strip -->
<rect x="80" y="378" width="1020" height="58" rx="12" fill="#c026d3" fill-opacity="0.05" stroke="#c026d3" stroke-width="1.5"></rect>
<text x="100" y="402" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12.5" font-weight="700" fill="#c026d3">absence fails loudly</text>
<text x="100" y="422" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11.5" fill="#0e1726">field required in the schema &#8594; a cluster without it fails at template time, in the pipeline &#8212; not months later as policy pointed at nothing</text>
</svg>

After

Width:  |  Height:  |  Size: 4.1 KiB

+79
View File
@@ -0,0 +1,79 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="520" viewBox="0 0 1180 520" role="img" aria-label="The probation ladder for downloaded models, left to right. Arrive: the artefact is more than weights - tokeniser code, an executable chat template, a loader gated by trust_remote_code - pinned by digest with its runtime. Probation tier: a Sandbox resource wraps a pod whose kata runtime class boots a per-pod KVM microVM with its own guest kernel, inside a zero-peer default-deny namespace holding zero credentials; the broker tier above holds all real keys. Observe: real workloads through the harness build a record of egress and behaviour against the written declaration. Gate: when the record matches the declaration over time the model promotes to the standard serving tier; unproven models stay in probation indefinitely. A hardware boundary separates the container from the node kernel for the whole supervised shift.">
<rect width="1180" height="520" fill="#ffffff"></rect>
<!-- title -->
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">The probation ladder</text>
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">a new model is a new hire &#8212; promotion is earned on the record, never granted on arrival</text>
<!-- spine -->
<line x1="80" y1="105" x2="1100" y2="105" stroke="#64748b" stroke-width="1.5"></line>
<circle cx="200" cy="105" r="5" fill="#7c3aed"></circle>
<circle cx="530" cy="105" r="5" fill="#0c8fce"></circle>
<circle cx="830" cy="105" r="5" fill="#64748b"></circle>
<circle cx="1020" cy="105" r="5" fill="#c026d3"></circle>
<!-- ARRIVE -->
<text x="200" y="140" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#7c3aed">ARRIVE</text>
<rect x="80" y="152" width="240" height="230" rx="14" fill="#7c3aed" fill-opacity="0.04" stroke="#7c3aed" stroke-width="2"></rect>
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">
<text x="100" y="182">not just weights:</text>
<text x="100" y="204">+ tokeniser code</text>
<text x="100" y="224">+ executable chat template</text>
<text x="100" y="244">+ loader (trust_remote_code)</text>
<text x="100" y="264">+ sometimes its own runtime</text>
</g>
<text x="100" y="300" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12" font-weight="700" fill="#7c3aed">pinned by digest,</text>
<text x="100" y="318" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12" font-weight="700" fill="#7c3aed">artefact + runtime</text>
<text x="100" y="352" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">the CV: an account name and</text>
<text x="100" y="368" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">a download count &#8212; uncheckable</text>
<!-- PROBATION (highlighted) -->
<text x="530" y="140" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#0c8fce">PROBATION &#8212; the supervised shift</text>
<rect x="360" y="152" width="340" height="230" rx="14" fill="#0c8fce" fill-opacity="0.07" stroke="#0c8fce" stroke-width="3"></rect>
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">
<text x="380" y="182">Sandbox CR &#8594; pod &#8594; runtimeClass kata</text>
<text x="380" y="204">&#8594; per-pod KVM microVM,</text>
<text x="392" y="222">its OWN guest kernel</text>
<text x="380" y="248">namespace: default-deny, zero peers</text>
<text x="380" y="268">credentials mounted: none</text>
<text x="380" y="288">egress: only what is declared</text>
<text x="380" y="308">keys: held by the broker tier above</text>
</g>
<line x1="380" y1="326" x2="680" y2="326" stroke="#0c8fce" stroke-width="1.5" stroke-dasharray="6 5"></line>
<text x="380" y="348" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" font-weight="700" fill="#0c8fce">hardware boundary: an escape lands in a guest kernel,</text>
<text x="380" y="366" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" font-weight="700" fill="#0c8fce">not on the node</text>
<!-- OBSERVE -->
<text x="830" y="140" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#64748b">OBSERVE</text>
<rect x="740" y="152" width="180" height="230" rx="14" fill="#64748b" fill-opacity="0.04" stroke="#64748b" stroke-width="2"></rect>
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#0e1726">
<text x="758" y="182">real workloads</text>
<text x="758" y="202">through the</text>
<text x="758" y="222">harness</text>
<text x="758" y="252">the record:</text>
<text x="758" y="272">egress log vs</text>
<text x="758" y="292">declaration</text>
<text x="758" y="312">behaviour vs</text>
<text x="758" y="332">expectation</text>
</g>
<text x="758" y="366" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">long enough, boring enough</text>
<!-- GATE -->
<text x="1020" y="140" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#c026d3">GATE</text>
<rect x="950" y="152" width="150" height="230" rx="14" fill="#c026d3" fill-opacity="0.05" stroke="#c026d3" stroke-width="2"></rect>
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11">
<text x="968" y="188" fill="#0c8fce">record matches</text>
<text x="968" y="208" fill="#0c8fce">&#8594; PROMOTE to</text>
<text x="980" y="228" fill="#0c8fce">serving tier</text>
<text x="968" y="266" fill="#c026d3">unproven</text>
<text x="968" y="286" fill="#c026d3">&#8594; stays in the</text>
<text x="980" y="306" fill="#c026d3">VM. forever</text>
<text x="980" y="326" fill="#c026d3">if need be</text>
</g>
<text x="968" y="362" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">permanent probation</text>
<text x="968" y="378" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11" fill="#64748b">costs almost nothing</text>
<!-- footer -->
<text x="590" y="452" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12.5" font-weight="700" fill="#0e1726">trust is a history, not a property &#8212; observed behaviour, under constraint, accumulated until it's boring</text>
<text x="590" y="478" text-anchor="middle" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11" fill="#64748b">the tax (seconds to boot, 350 MiB per microVM, batch-shaped work) isn't a flaw in the ladder &#8212; the tax IS the ladder</text>
</svg>

After

Width:  |  Height:  |  Size: 7.3 KiB

+44
View File
@@ -0,0 +1,44 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1180" height="500" viewBox="0 0 1180 500" role="img" aria-label="Two views of the same six days. What the platform saw: the container running, the process alive, the port answering, logs present - every default check passing. What was true: the tunnel the workload exists to use was down, nothing had moved for six days, and the retry backoff made the logs quieter each day. A timeline underneath runs from the failure on day zero to discovery on day six - found by a person asking a question, not by the stack. The fix strip: probe the capability, not the container - the tunnel's own health check gates the workload, so health only reports when traffic genuinely egresses.">
<rect width="1180" height="500" fill="#ffffff"></rect>
<!-- title -->
<text x="60" y="46" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="23" font-weight="700" fill="#0e1726">Running was never the job</text>
<text x="61" y="70" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="13.5" fill="#64748b">the gap between &#34;the process is running&#34; and &#34;the process is doing its job&#34; is where invisible outages live</text>
<!-- platform view -->
<text x="315" y="112" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#0c8fce">WHAT THE PLATFORM SAW</text>
<rect x="70" y="126" width="490" height="170" rx="14" fill="#0c8fce" fill-opacity="0.05" stroke="#0c8fce" stroke-width="2"></rect>
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12" fill="#0e1726">
<text x="100" y="160">container: Running</text>
<text x="100" y="184">process: alive, answers, holds its port</text>
<text x="100" y="208">every default check: passing</text>
<text x="100" y="232">logs: present &#8212; and getting quieter</text>
</g>
<text x="100" y="272" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">entirely satisfied &#183; no reason to think otherwise</text>
<!-- truth view -->
<text x="865" y="112" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="15" font-weight="700" fill="#c026d3">WHAT WAS TRUE</text>
<rect x="620" y="126" width="490" height="170" rx="14" fill="#c026d3" fill-opacity="0.04" stroke="#c026d3" stroke-width="2"></rect>
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="12" fill="#0e1726">
<text x="650" y="160">the tunnel: down</text>
<text x="650" y="184">traffic moved: none, for six days</text>
<text x="650" y="208">retry backoff: growing &#8212; the failure</text>
<text x="662" y="230">was quietly obscuring itself</text>
</g>
<text x="650" y="272" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">alive, responsive, and completely useless</text>
<!-- timeline -->
<line x1="90" y1="340" x2="1090" y2="340" stroke="#64748b" stroke-width="1.5"></line>
<circle cx="130" cy="340" r="6" fill="#c026d3"></circle>
<circle cx="1050" cy="340" r="6" fill="#0c8fce"></circle>
<g font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11.5">
<text x="130" y="322" text-anchor="middle" fill="#c026d3">day 0: tunnel dies</text>
<text x="1050" y="322" text-anchor="middle" fill="#0c8fce">day 6: a person asks</text>
</g>
<text x="590" y="366" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="11.5" fill="#64748b">six days of nothing &#8212; discovered by accident, the way every unwatched failure is discovered</text>
<!-- fix strip -->
<rect x="70" y="396" width="1040" height="70" rx="12" fill="#0c8fce" fill-opacity="0.05" stroke="#0c8fce" stroke-width="1.5"></rect>
<text x="90" y="422" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" font-size="12.5" font-weight="700" fill="#0c8fce">probe the capability, not the container</text>
<text x="90" y="444" font-family="ui-monospace, 'SF Mono', Menlo, monospace" font-size="11.5" fill="#0e1726">the tunnel's own health check gates the workload &#8594; passes only when traffic genuinely egresses &#8594; silent failure becomes a restart, then an alert</text>
</svg>

After

Width:  |  Height:  |  Size: 4.4 KiB