diff --git a/src/components/About.astro b/src/components/About.astro index d8ec233..4fa60e8 100644 --- a/src/components/About.astro +++ b/src/components/About.astro @@ -23,7 +23,7 @@ const olderCount = experience.length - recent.length; -

# education

+

# education

@@ -100,7 +100,7 @@ const olderCount = experience.length - recent.length; font-size: var(--step--1); letter-spacing: 0.02em; } - .about__sub::first-letter { + .about__hash { color: var(--accent); } .about__education { diff --git a/src/content/projects/edge-ai-platform.md b/src/content/projects/edge-ai-platform.md index b9822af..0945584 100644 --- a/src/content/projects/edge-ai-platform.md +++ b/src/content/projects/edge-ai-platform.md @@ -1,7 +1,7 @@ --- title: "Single-Touch Edge AI Platform" outcome: "Turned a high-level edge-AI design into a single-touch deployment running on Kubernetes at the store edge." -summary: "Store-edge Kubernetes running GPU-backed AI workloads, deployed from one command, with readiness-gated GPUs so inference never starts before the hardware is ready." +summary: "Store-edge Kubernetes running GPU-backed AI workloads, deployed from one command, with an init gate that holds the inference container until the GPU answers." role: "Infrastructure / DevOps Engineer ยท Woolworths" period: "2025 - Present" stack: ["Kubernetes", "Edge", "NVIDIA GPU", "CD pipelines", "Helm", "Python"] @@ -22,8 +22,9 @@ the gap between "it works" and "it ships." ## Constraints - **No hands at the edge.** Deployment has to be hands-off and idempotent - a single press. -- **GPU timing.** Inference pods must never schedule before the GPU device plugin is healthy, - or they crash-loop and poison the rollout. +- **GPU timing.** The kubelet can report `Ready` before the NVIDIA device plugin has advertised + `nvidia.com/gpu`. An inference container that starts in that window crash-loops and poisons + the rollout. - **Heterogeneous stores.** Per-site variables (network, hardware, identity) without forking the platform for every location. @@ -35,15 +36,17 @@ store-edge Kubernetes cluster via Helm with end-state manifests. Per-store confi injected from a single source of truth, so one pipeline produces a correct deployment for any site. -The load-bearing piece is **readiness gating**: Bash/Shell probes and Kubernetes watchdogs -confirm the GPU device plugin is up *before* inference pods are allowed to run, and pod -lifecycle management keeps the workload honest from there. +Three separate controls do three separate jobs. A `nvidia.com/gpu` resource request decides +*where* the pod lands. An **init container** blocks until `nvidia-smi` enumerates a device, so the +inference container cannot start before the hardware answers - and it fails loudly, with a bound, +rather than waiting forever. Runtime probes and watchdogs then track health *after* start. ## Security & reliability decisions -- **Init-gated GPU readiness** - the single biggest reliability win; no more pods racing the - GPU at boot. -- **Single source of truth** for config - drift can't creep in store-to-store. +- **Init-gated GPU startup** - the single biggest reliability win; the app container no longer + races the device plugin at boot. +- **Single source of truth** for config, rendered through templates - the class of divergence + where two stores disagree on the same value fails at render time rather than in production. - **Spec-driven, documented-as-code** - the deployment *is* the documentation. ## Outcome diff --git a/src/content/projects/gpu-as-code.md b/src/content/projects/gpu-as-code.md index d23d090..ac696a9 100644 --- a/src/content/projects/gpu-as-code.md +++ b/src/content/projects/gpu-as-code.md @@ -1,7 +1,7 @@ --- title: "GPU-as-Code on the Edge" -outcome: "Brought GPUs online as code - passthrough, readiness-gated, and reproducible across the fleet." -summary: "GPU passthrough configured through ESXi via code with end-state manifests and Helm, paired with readiness probes, watchdogs, and DCGM-based health reporting." +outcome: "Brought GPUs online as code - passthrough, init-gated at startup, and reproducible across the fleet." +summary: "GPU passthrough configured through ESXi via code with end-state manifests and Helm, paired with an init gate on GPU availability, runtime watchdogs, and DCGM-based health reporting." role: "Infrastructure / DevOps Engineer" period: "2025 - Present" stack: ["GPU passthrough", "ESXi", "DCGM Exporter", "Prometheus", "Bash", "Watchdogs"] @@ -25,22 +25,23 @@ to refuse to start until both are true. Doing that by hand, per site, doesn't sc ## Design GPU passthrough is configured through ESXi **via code**, with end-state manifests and Helm -charts describing the desired node. In-cluster, **Bash/Shell readiness probes** and -Kubernetes **watchdogs** gate inference pods on a healthy GPU device plugin and manage pod -lifecycle from there. **DCGM Exporter** feeds GPU and container-workload health into +charts describing the desired node. In-cluster, an **init container** holds the inference +container until the GPU enumerates, and **runtime probes and watchdogs** track health from +there - startup gating and health checking kept as separate concerns. **DCGM Exporter** feeds GPU and container-workload health into Prometheus and AWX job-level reporting, so a degraded GPU surfaces the same way any other platform signal does. ## Security & reliability decisions -- **Readiness gating** - pods wait for the hardware; no boot-time races. +- **Init gating** - the app container waits for the hardware, so there are no boot-time races. - **End-state manifests** - the node's GPU config is declarative and reproducible. - **DCGM telemetry** - GPU failures are detected, not discovered. ## Outcome GPUs come online predictably across the fleet, the dangerous "pod started before the GPU" -class of failure is designed out, and GPU health is a first-class metric. +class of failure is addressed at startup rather than left to retries, and GPU health is +reported alongside every other platform signal. ## Future improvements diff --git a/src/content/projects/network-automation-fleet.md b/src/content/projects/network-automation-fleet.md index 469e353..ca74c57 100644 --- a/src/content/projects/network-automation-fleet.md +++ b/src/content/projects/network-automation-fleet.md @@ -42,15 +42,16 @@ Paired with it, **NetBox** served as the **IPAM and source-of-truth** for the de inventory the automation runs against - so the fleet is driven from a single, accurate model of what exists rather than tribal knowledge. Both are lean, community-rooted tools adopted under a **corporate-supported licence**, which made them viable inside an enterprise change -process - the open-source-into-corporate pattern, done properly. +process, under a corporate-supported licence so the tool had a support path. ## Security & reliability decisions - **Centralised, vaulted credentials with rotation** - the network's secrets live in one scoped, rotated place, never embedded in scripts or shared by hand. -- **Config backup = real network DR** - every device restorable from a known-good config. -- **Diffs + auditing** - every configuration change is visible, attributable, and reversible. -- **Source-of-truth driven** - automation acts on NetBox's model, not on guesswork. +- **Known-good config backups** - a recovery path for device configuration. Full network DR also depends on replacement hardware, firmware and console access, which sit outside this platform. +- **Diffs + auditing** - configuration changes are visible and attributable, and the stored known-good config is the material you roll back from. Changes that cut management access or touch firmware still need console/physical access. +- **Source-of-truth driven** - NetBox holds the authoritative device and address model the automation is run against. + ## Outcome diff --git a/src/content/projects/self-hosted-ai-homelab.md b/src/content/projects/self-hosted-ai-homelab.md index d71849a..c503900 100644 --- a/src/content/projects/self-hosted-ai-homelab.md +++ b/src/content/projects/self-hosted-ai-homelab.md @@ -1,6 +1,6 @@ --- title: "Self-Hosted AI & Homelab Platform" -outcome: "A production-grade homelab - GitOps from bare metal to local AI, and the platform that serves this very site." +outcome: "A homelab - GitOps from bare metal to local AI, and the platform that serves this very site." summary: "Proxmox with PCIe passthrough under Talos and OpenShift clusters, all driven by Argo CD GitOps: local LLM inference, split-horizon DNS, 2FA/SSO VPN, full observability and NAS-backed backups." role: "Owner / Operator" period: "Ongoing" diff --git a/src/data/skills.ts b/src/data/skills.ts index 6604beb..cf40501 100644 --- a/src/data/skills.ts +++ b/src/data/skills.ts @@ -25,7 +25,7 @@ export const skills: SkillGroup[] = [ blurb: "Serving vision and language models on real hardware at the edge.", points: [ "Fail-closed computer vision on single-node OpenShift - probes that catch a silently dead model, not just a dead pod.", - "One GPU, two jobs - a single card time-sliced between LLM serving and video transcode, with neither noticing.", + "One GPU, two workloads - a single card time-sliced between LLM serving and video transcode instead of reserving separate hardware.", "Vendor video-analytics stacks re-platformed as first-class Kubernetes workloads, not appliances.", "Local models behind an OpenAI-compatible gateway - swap the backend, keep every client.", ],