second pass: name the real GPU mechanism (resource request schedules, init container gates startup, probes track runtime health), scope the network DR and reversibility claims to what config backup actually gives you, drop production-grade, and colour the education hash instead of the first letter
build-and-deploy / build (push) Failing after 10m52s
build-and-deploy / build (push) Failing after 10m52s
This commit is contained in:
@@ -23,7 +23,7 @@ const olderCount = experience.length - recent.length;
|
|||||||
</div>
|
</div>
|
||||||
</dl>
|
</dl>
|
||||||
|
|
||||||
<h3 class="about__sub mono"># education</h3>
|
<h3 class="about__sub mono"><span class="about__hash">#</span> education</h3>
|
||||||
<ul class="about__education">
|
<ul class="about__education">
|
||||||
{education.map((e) => <li>{e}</li>)}
|
{education.map((e) => <li>{e}</li>)}
|
||||||
</ul>
|
</ul>
|
||||||
@@ -100,7 +100,7 @@ const olderCount = experience.length - recent.length;
|
|||||||
font-size: var(--step--1);
|
font-size: var(--step--1);
|
||||||
letter-spacing: 0.02em;
|
letter-spacing: 0.02em;
|
||||||
}
|
}
|
||||||
.about__sub::first-letter {
|
.about__hash {
|
||||||
color: var(--accent);
|
color: var(--accent);
|
||||||
}
|
}
|
||||||
.about__education {
|
.about__education {
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
title: "Single-Touch Edge AI Platform"
|
title: "Single-Touch Edge AI Platform"
|
||||||
outcome: "Turned a high-level edge-AI design into a single-touch deployment running on Kubernetes at the store edge."
|
outcome: "Turned a high-level edge-AI design into a single-touch deployment running on Kubernetes at the store edge."
|
||||||
summary: "Store-edge Kubernetes running GPU-backed AI workloads, deployed from one command, with readiness-gated GPUs so inference never starts before the hardware is ready."
|
summary: "Store-edge Kubernetes running GPU-backed AI workloads, deployed from one command, with an init gate that holds the inference container until the GPU answers."
|
||||||
role: "Infrastructure / DevOps Engineer · Woolworths"
|
role: "Infrastructure / DevOps Engineer · Woolworths"
|
||||||
period: "2025 - Present"
|
period: "2025 - Present"
|
||||||
stack: ["Kubernetes", "Edge", "NVIDIA GPU", "CD pipelines", "Helm", "Python"]
|
stack: ["Kubernetes", "Edge", "NVIDIA GPU", "CD pipelines", "Helm", "Python"]
|
||||||
@@ -22,8 +22,9 @@ the gap between "it works" and "it ships."
|
|||||||
## Constraints
|
## Constraints
|
||||||
|
|
||||||
- **No hands at the edge.** Deployment has to be hands-off and idempotent - a single press.
|
- **No hands at the edge.** Deployment has to be hands-off and idempotent - a single press.
|
||||||
- **GPU timing.** Inference pods must never schedule before the GPU device plugin is healthy,
|
- **GPU timing.** The kubelet can report `Ready` before the NVIDIA device plugin has advertised
|
||||||
or they crash-loop and poison the rollout.
|
`nvidia.com/gpu`. An inference container that starts in that window crash-loops and poisons
|
||||||
|
the rollout.
|
||||||
- **Heterogeneous stores.** Per-site variables (network, hardware, identity) without forking
|
- **Heterogeneous stores.** Per-site variables (network, hardware, identity) without forking
|
||||||
the platform for every location.
|
the platform for every location.
|
||||||
|
|
||||||
@@ -35,15 +36,17 @@ store-edge Kubernetes cluster via Helm with end-state manifests. Per-store confi
|
|||||||
injected from a single source of truth, so one pipeline produces a correct deployment for
|
injected from a single source of truth, so one pipeline produces a correct deployment for
|
||||||
any site.
|
any site.
|
||||||
|
|
||||||
The load-bearing piece is **readiness gating**: Bash/Shell probes and Kubernetes watchdogs
|
Three separate controls do three separate jobs. A `nvidia.com/gpu` resource request decides
|
||||||
confirm the GPU device plugin is up *before* inference pods are allowed to run, and pod
|
*where* the pod lands. An **init container** blocks until `nvidia-smi` enumerates a device, so the
|
||||||
lifecycle management keeps the workload honest from there.
|
inference container cannot start before the hardware answers - and it fails loudly, with a bound,
|
||||||
|
rather than waiting forever. Runtime probes and watchdogs then track health *after* start.
|
||||||
|
|
||||||
## Security & reliability decisions
|
## Security & reliability decisions
|
||||||
|
|
||||||
- **Init-gated GPU readiness** - the single biggest reliability win; no more pods racing the
|
- **Init-gated GPU startup** - the single biggest reliability win; the app container no longer
|
||||||
GPU at boot.
|
races the device plugin at boot.
|
||||||
- **Single source of truth** for config - drift can't creep in store-to-store.
|
- **Single source of truth** for config, rendered through templates - the class of divergence
|
||||||
|
where two stores disagree on the same value fails at render time rather than in production.
|
||||||
- **Spec-driven, documented-as-code** - the deployment *is* the documentation.
|
- **Spec-driven, documented-as-code** - the deployment *is* the documentation.
|
||||||
|
|
||||||
## Outcome
|
## Outcome
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
title: "GPU-as-Code on the Edge"
|
title: "GPU-as-Code on the Edge"
|
||||||
outcome: "Brought GPUs online as code - passthrough, readiness-gated, and reproducible across the fleet."
|
outcome: "Brought GPUs online as code - passthrough, init-gated at startup, and reproducible across the fleet."
|
||||||
summary: "GPU passthrough configured through ESXi via code with end-state manifests and Helm, paired with readiness probes, watchdogs, and DCGM-based health reporting."
|
summary: "GPU passthrough configured through ESXi via code with end-state manifests and Helm, paired with an init gate on GPU availability, runtime watchdogs, and DCGM-based health reporting."
|
||||||
role: "Infrastructure / DevOps Engineer"
|
role: "Infrastructure / DevOps Engineer"
|
||||||
period: "2025 - Present"
|
period: "2025 - Present"
|
||||||
stack: ["GPU passthrough", "ESXi", "DCGM Exporter", "Prometheus", "Bash", "Watchdogs"]
|
stack: ["GPU passthrough", "ESXi", "DCGM Exporter", "Prometheus", "Bash", "Watchdogs"]
|
||||||
@@ -25,22 +25,23 @@ to refuse to start until both are true. Doing that by hand, per site, doesn't sc
|
|||||||
## Design
|
## Design
|
||||||
|
|
||||||
GPU passthrough is configured through ESXi **via code**, with end-state manifests and Helm
|
GPU passthrough is configured through ESXi **via code**, with end-state manifests and Helm
|
||||||
charts describing the desired node. In-cluster, **Bash/Shell readiness probes** and
|
charts describing the desired node. In-cluster, an **init container** holds the inference
|
||||||
Kubernetes **watchdogs** gate inference pods on a healthy GPU device plugin and manage pod
|
container until the GPU enumerates, and **runtime probes and watchdogs** track health from
|
||||||
lifecycle from there. **DCGM Exporter** feeds GPU and container-workload health into
|
there - startup gating and health checking kept as separate concerns. **DCGM Exporter** feeds GPU and container-workload health into
|
||||||
Prometheus and AWX job-level reporting, so a degraded GPU surfaces the same way any other
|
Prometheus and AWX job-level reporting, so a degraded GPU surfaces the same way any other
|
||||||
platform signal does.
|
platform signal does.
|
||||||
|
|
||||||
## Security & reliability decisions
|
## Security & reliability decisions
|
||||||
|
|
||||||
- **Readiness gating** - pods wait for the hardware; no boot-time races.
|
- **Init gating** - the app container waits for the hardware, so there are no boot-time races.
|
||||||
- **End-state manifests** - the node's GPU config is declarative and reproducible.
|
- **End-state manifests** - the node's GPU config is declarative and reproducible.
|
||||||
- **DCGM telemetry** - GPU failures are detected, not discovered.
|
- **DCGM telemetry** - GPU failures are detected, not discovered.
|
||||||
|
|
||||||
## Outcome
|
## Outcome
|
||||||
|
|
||||||
GPUs come online predictably across the fleet, the dangerous "pod started before the GPU"
|
GPUs come online predictably across the fleet, the dangerous "pod started before the GPU"
|
||||||
class of failure is designed out, and GPU health is a first-class metric.
|
class of failure is addressed at startup rather than left to retries, and GPU health is
|
||||||
|
reported alongside every other platform signal.
|
||||||
|
|
||||||
## Future improvements
|
## Future improvements
|
||||||
|
|
||||||
|
|||||||
@@ -42,15 +42,16 @@ Paired with it, **NetBox** served as the **IPAM and source-of-truth** for the de
|
|||||||
inventory the automation runs against - so the fleet is driven from a single, accurate model
|
inventory the automation runs against - so the fleet is driven from a single, accurate model
|
||||||
of what exists rather than tribal knowledge. Both are lean, community-rooted tools adopted
|
of what exists rather than tribal knowledge. Both are lean, community-rooted tools adopted
|
||||||
under a **corporate-supported licence**, which made them viable inside an enterprise change
|
under a **corporate-supported licence**, which made them viable inside an enterprise change
|
||||||
process - the open-source-into-corporate pattern, done properly.
|
process, under a corporate-supported licence so the tool had a support path.
|
||||||
|
|
||||||
## Security & reliability decisions
|
## Security & reliability decisions
|
||||||
|
|
||||||
- **Centralised, vaulted credentials with rotation** - the network's secrets live in one
|
- **Centralised, vaulted credentials with rotation** - the network's secrets live in one
|
||||||
scoped, rotated place, never embedded in scripts or shared by hand.
|
scoped, rotated place, never embedded in scripts or shared by hand.
|
||||||
- **Config backup = real network DR** - every device restorable from a known-good config.
|
- **Known-good config backups** - a recovery path for device configuration. Full network DR also depends on replacement hardware, firmware and console access, which sit outside this platform.
|
||||||
- **Diffs + auditing** - every configuration change is visible, attributable, and reversible.
|
- **Diffs + auditing** - configuration changes are visible and attributable, and the stored known-good config is the material you roll back from. Changes that cut management access or touch firmware still need console/physical access.
|
||||||
- **Source-of-truth driven** - automation acts on NetBox's model, not on guesswork.
|
- **Source-of-truth driven** - NetBox holds the authoritative device and address model the automation is run against.
|
||||||
|
<!-- TODO(Jonathon): confirm whether NetBox programmatically drove Unimus device inventory, or whether the two were complementary but not coupled. Wording is deliberately non-committal until confirmed. -->
|
||||||
|
|
||||||
## Outcome
|
## Outcome
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: "Self-Hosted AI & Homelab Platform"
|
title: "Self-Hosted AI & Homelab Platform"
|
||||||
outcome: "A production-grade homelab - GitOps from bare metal to local AI, and the platform that serves this very site."
|
outcome: "A homelab - GitOps from bare metal to local AI, and the platform that serves this very site."
|
||||||
summary: "Proxmox with PCIe passthrough under Talos and OpenShift clusters, all driven by Argo CD GitOps: local LLM inference, split-horizon DNS, 2FA/SSO VPN, full observability and NAS-backed backups."
|
summary: "Proxmox with PCIe passthrough under Talos and OpenShift clusters, all driven by Argo CD GitOps: local LLM inference, split-horizon DNS, 2FA/SSO VPN, full observability and NAS-backed backups."
|
||||||
role: "Owner / Operator"
|
role: "Owner / Operator"
|
||||||
period: "Ongoing"
|
period: "Ongoing"
|
||||||
|
|||||||
+1
-1
@@ -25,7 +25,7 @@ export const skills: SkillGroup[] = [
|
|||||||
blurb: "Serving vision and language models on real hardware at the edge.",
|
blurb: "Serving vision and language models on real hardware at the edge.",
|
||||||
points: [
|
points: [
|
||||||
"Fail-closed computer vision on single-node OpenShift - probes that catch a silently dead model, not just a dead pod.",
|
"Fail-closed computer vision on single-node OpenShift - probes that catch a silently dead model, not just a dead pod.",
|
||||||
"One GPU, two jobs - a single card time-sliced between LLM serving and video transcode, with neither noticing.",
|
"One GPU, two workloads - a single card time-sliced between LLM serving and video transcode instead of reserving separate hardware.",
|
||||||
"Vendor video-analytics stacks re-platformed as first-class Kubernetes workloads, not appliances.",
|
"Vendor video-analytics stacks re-platformed as first-class Kubernetes workloads, not appliances.",
|
||||||
"Local models behind an OpenAI-compatible gateway - swap the backend, keep every client.",
|
"Local models behind an OpenAI-compatible gateway - swap the backend, keep every client.",
|
||||||
],
|
],
|
||||||
|
|||||||
Reference in New Issue
Block a user