M2: content collections — case studies, blog, RSS, tags, sitemap

Projects + blog as schema-validated content collections; structured case studies (problem/design/outcome), blog with tag pages, reading time, RSS feed (drafts excluded), sitemap, and Shiki dual-theme code highlighting.
2026-06-17 16:56:46 +10:00
parent 720d579386
commit 22f482d89a
26 changed files with 1139 additions and 105 deletions
@@ -0,0 +1,11 @@
+---
+title: "Draft: notes on air-gapped registry mirroring"
+date: 2026-06-17
+summary: "Work in progress — this draft should never appear in the production build or the RSS feed."
+tags: ["draft", "registry"]
+draft: true
+---
+
+This post is intentionally marked `draft: true` to verify that drafts are excluded from the
+production build and the RSS feed. If you can read this on the live site, the draft filter is
+broken.
@@ -0,0 +1,52 @@
+---
+title: "Init-gating GPU readiness on Kubernetes"
+date: 2026-06-10
+summary: "The single highest-leverage reliability fix for edge GPU workloads: never let an inference pod schedule before the GPU is actually ready."
+tags: ["kubernetes", "gpu", "edge", "reliability"]
+---
+
+The most common way a GPU workload fails at the edge isn't the model, the driver, or the
+network. It's timing. Kubernetes is eager — it will happily schedule your inference pod the
+moment a node is `Ready`, which is often *before* the NVIDIA device plugin has advertised
+`nvidia.com/gpu`. The pod starts, can't see a GPU, crash-loops, and now your rollout is
+poisoned across the fleet.
+
+The fix is to make readiness explicit. Don't trust node-`Ready`; gate on the GPU.
+
+## Gate the schedule, not just the start
+
+A resource request is the first line — a pod that *requests* a GPU won't schedule until the
+plugin advertises capacity:
+
+```yaml
+resources:
+  limits:
+    nvidia.com/gpu: 1
+```
+
+But on a single-GPU edge node that's recovering from a reboot, you still want a hard check
+before the workload does anything expensive. An init container that blocks until the device
+is real keeps the main container honest:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+# Block until the GPU is visible AND healthy, or fail loudly after a bound.
+for i in $(seq 1 30); do
+  if nvidia-smi -L | grep -q '^GPU 0'; then
+    echo "GPU ready"; exit 0
+  fi
+  echo "waiting for GPU ($i/30)"; sleep 5
+done
+echo "GPU never became ready" >&2
+exit 1
+```
+
+## Why this is the win
+
+Once readiness is gated, the whole class of "pod started before the GPU" failures disappears
+— and it disappears *the same way on every node*. That consistency is the real prize at the
+edge, where no one is standing next to the box to nurse a bad rollout.
+
+The principle generalises: at the edge, **design the dependency, don't hope for it**. The GPU
+is just the first dependency worth gating; egress paths and model artifacts are next.
@@ -0,0 +1,40 @@
+---
+title: "Shipping this site: GitOps from a homelab to the public internet"
+date: 2026-06-15
+summary: "How this portfolio is built and served — Astro to a container image, a self-hosted Gitea registry, ArgoCD, and a Cloudflare Tunnel — with security as acceptance criteria, not polish."
+tags: ["gitops", "astro", "homelab", "security"]
+---
+
+This site is a static Astro build, but how it gets to you is the interesting part. It's
+served from my homelab Kubernetes cluster over a Cloudflare Tunnel, deployed the same way I'd
+ship anything else: as an immutable image, pinned by digest, reconciled by GitOps.
+
+## The pipeline
+
+1. The site is built and baked into a hardened `nginx-unprivileged` image.
+2. The image is pushed to a **self-hosted public Gitea registry** — deliberately separate
+   from the private instance that holds my infrastructure code.
+3. The image digest is pinned in a private `home-ops` repo.
+4. **ArgoCD** reconciles that repo onto the cluster.
+5. A **Cloudflare Tunnel** exposes exactly one service — this site — outbound-only.
+
+No open ports. No server runtime. No registry credential on the cluster, because the public
+package is anonymous-pull and the image holds nothing secret.
+
+## Security as acceptance criteria
+
+The interesting constraint was treating security as a checklist to *pass*, not a vibe:
+
+```text
+[x] Static output — no server runtime to attack
+[x] Strict CSP, no unsafe-inline / unsafe-eval
+[x] Self-hosted fonts — zero third-party requests
+[x] No secrets in the client bundle (verified by build-time grep)
+[x] Outbound-only tunnel, single hostname, no catch-all
+```
+
+## Why bother
+
+Because the site *is* the argument. A platform engineer's portfolio should demonstrate the
+discipline it's advertising — and "it's a static page" is no excuse to skip the rigour. The
+deployment story is part of the work.
@@ -0,0 +1,58 @@
+---
+title: "Single-Touch Edge AI Platform"
+outcome: "Turned a high-level edge-AI design into a single-press deployment running on Kubernetes at the store edge."
+summary: "Store-edge Kubernetes running GPU-backed AI workloads, deployed from one command, with readiness-gated GPUs so inference never starts before the hardware is ready."
+role: "Infrastructure / DevOps Engineer · Woolworths"
+period: "2025 – Present"
+stack: ["Kubernetes", "Edge", "NVIDIA GPU", "CD pipelines", "Helm", "Python"]
+featured: true
+order: 10
+diagram: "edge-ai"
+---
+
+## Problem
+
+Edge AI at retail scale lives or dies on repeatability. A computer-vision workload that
+runs perfectly in a lab has to come up the same way in a store with no on-site engineer,
+flaky connectivity, and a GPU that may not be ready the instant Kubernetes wants to schedule
+against it. The starting point was a high-level design and a pile of manual steps — exactly
+the gap between "it works" and "it ships."
+
+## Constraints
+
+- **No hands at the edge.** Deployment has to be hands-off and idempotent — a single press.
+- **GPU timing.** Inference pods must never schedule before the GPU device plugin is healthy,
+  or they crash-loop and poison the rollout.
+- **Heterogeneous stores.** Per-site variables (network, hardware, identity) without forking
+  the platform for every location.
+
+## Design
+
+I took the high-level designs and turned them into low-level, problem-solving deployments
+driven by CD pipelines. The application is packaged as containers and shipped to a
+store-edge Kubernetes cluster via Helm with end-state manifests. Per-store configuration is
+injected from a single source of truth, so one pipeline produces a correct deployment for
+any site.
+
+The load-bearing piece is **readiness gating**: Bash/Shell probes and Kubernetes watchdogs
+confirm the GPU device plugin is up *before* inference pods are allowed to run, and pod
+lifecycle management keeps the workload honest from there.
+
+## Security & reliability decisions
+
+- **Init-gated GPU readiness** — the single biggest reliability win; no more pods racing the
+  GPU at boot.
+- **Single source of truth** for config — drift can't creep in store-to-store.
+- **Spec-driven, documented-as-code** — the deployment *is* the documentation.
+
+## Outcome
+
+A high-level idea becomes a real, repeatable deployment on a single press. New edge sites
+come up consistently, GPUs come online reliably, and the manual runbook is gone — replaced
+by a pipeline anyone on the team can trigger.
+
+## Future improvements
+
+Push more of the per-store delta into declarative policy, and extend the readiness model to
+cover the full inference dependency chain (model artifacts, egress, downstream sinks) as a
+single health gate.
@@ -0,0 +1,49 @@
+---
+title: "Global Infrastructure Modernisation"
+outcome: "Modernised enterprise infrastructure at scale — ~1,000 VMs, segmented networks, multi-region cloud migration."
+summary: "Across global IT roles: a ~1,000-VM VMware estate, flat-to-segmented network redesign with SD-WAN and Aruba ClearPass, firewall upgrades, and migration to Azure and Microsoft 365."
+role: "Infrastructure Engineer · Virtus Health / Linde"
+period: "2019 – 2025"
+stack: ["VMware", "Azure", "SD-WAN", "Aruba ClearPass", "FortiGate", "Microsoft 365"]
+featured: false
+order: 40
+---
+
+## Problem
+
+Enterprise estates accrete. Flat networks, sprawling VM counts, aging firewalls, and
+on-prem-only services become a security and operations drag. The work: modernise without
+breaking a global business that runs 24/7.
+
+## Constraints
+
+- **Keep the lights on** — change a live, multi-region estate without downtime.
+- **Security and compliance** — segmentation, patching, and auditability throughout.
+- **Cost-aware** — modernise to cloud where it pays, not for its own sake.
+
+## Design
+
+Across global roles I ran and improved a **~1,000-VM VMware estate** and re-segmented **flat
+sites into isolated VLAN ranges**, layering in **SD-WAN** and **Aruba ClearPass** onboarding
+for a tiered, authenticated network. **Palo Alto / FortiGate** firewalls were upgraded and
+redesigned around the new segmentation. Workloads and identity moved to **Azure** (Blob, AVS)
+and **Microsoft 365** — including an ERP hardware refresh with a new DR solution, and a
+region-wide PBX-to-VoIP migration.
+
+## Security & reliability decisions
+
+- **Flat → segmented** — isolation by design, not by exception.
+- **Authenticated access** (ClearPass, 802.1x) — the network knows who's on it.
+- **Patched, current firewalls** — closing the easy doors first.
+- **DR built in** — recovery designed, not assumed.
+
+## Outcome
+
+A more secure, segmented, cloud-leaning estate that's cheaper to run and easier to operate —
+delivered against live-business constraints across multiple regions.
+
+## Future improvements
+
+The throughline from this work to the edge platforms: take the same segmentation and
+identity rigour and express it as code, so a thousand-VM estate and a single edge node are
+governed the same way.
@@ -0,0 +1,47 @@
+---
+title: "GPU-as-Code on the Edge"
+outcome: "Brought GPUs online as code — passthrough, readiness-gated, and reproducible across the fleet."
+summary: "GPU passthrough configured through ESXi via code with end-state manifests and Helm, paired with readiness probes, watchdogs, and DCGM-based health reporting."
+role: "Infrastructure / DevOps Engineer"
+period: "2025 – Present"
+stack: ["GPU passthrough", "ESXi", "DCGM Exporter", "Prometheus", "Bash", "Watchdogs"]
+featured: false
+order: 30
+---
+
+## Problem
+
+GPUs are the most failure-prone part of an edge AI stack: passthrough has to be configured
+on the hypervisor, the device plugin has to be healthy in the cluster, and the workload has
+to refuse to start until both are true. Doing that by hand, per site, doesn't scale.
+
+## Constraints
+
+- **As-code, not click-ops** — GPU passthrough defined in code, not the ESXi UI.
+- **Fail safe** — a not-ready GPU must block the workload, not crash it.
+- **Observable** — GPU health has to be visible alongside the rest of the platform.
+
+## Design
+
+GPU passthrough is configured through ESXi **via code**, with end-state manifests and Helm
+charts describing the desired node. In-cluster, **Bash/Shell readiness probes** and
+Kubernetes **watchdogs** gate inference pods on a healthy GPU device plugin and manage pod
+lifecycle from there. **DCGM Exporter** feeds GPU and container-workload health into
+Prometheus and AWX job-level reporting, so a degraded GPU surfaces the same way any other
+platform signal does.
+
+## Security & reliability decisions
+
+- **Readiness gating** — pods wait for the hardware; no boot-time races.
+- **End-state manifests** — the node's GPU config is declarative and reproducible.
+- **DCGM telemetry** — GPU failures are detected, not discovered.
+
+## Outcome
+
+GPUs come online predictably across the fleet, the dangerous "pod started before the GPU"
+class of failure is designed out, and GPU health is a first-class metric.
+
+## Future improvements
+
+Roll the readiness contract and DCGM thresholds into a single reusable module so any new
+GPU workload inherits the same guarantees by default.
@@ -0,0 +1,51 @@
+---
+title: "IaC Fleet Automation"
+outcome: "Stood up identical edge sites from code — every store comes up the same way, every time."
+summary: "Ansible/AWX playbooks wired through a single source-of-truth pipeline: GPU operator, templated networking, image pre-pull and secrets — with air-gapped registry mirroring for disconnected sites."
+role: "Automation Engineer"
+period: "2025 – Present"
+stack: ["Ansible", "AWX", "GitOps", "ACR / NVCR", "Image pre-pull", "Secrets mgmt"]
+featured: true
+order: 20
+diagram: "iac-fleet"
+---
+
+## Problem
+
+A fleet only behaves like a fleet if every node is built the same way. Hand-configuring GPU
+drivers, CNI, image caches and secrets per site is slow, error-prone, and impossible to
+audit — and at the edge, half the sites can't reach the internet when you need them to.
+
+## Constraints
+
+- **Repeatability over cleverness** — the same playbook must produce the same node anywhere.
+- **Air-gapped reality** — disconnected edge sites still have to build from local images.
+- **No secrets in code** — credentials delivered at deploy time, never committed.
+
+## Design
+
+Ansible playbooks, orchestrated by AWX and wired through a single source-of-truth pipeline,
+own the whole node build: GPU operator install, templated network attachments, container
+**image pre-pull**, and secrets injected from a managed store. Company- and site-specific
+variables are layered on top of a shared base so one playbook set serves the whole fleet.
+
+For disconnected sites, **air-gapped registry workflows** mirror images across Azure
+Container Registry and NVCR and pre-stage them locally, so a build never depends on a live
+internet path at the moment it matters.
+
+## Security & reliability decisions
+
+- **Secrets management at deploy time** — nothing sensitive in git.
+- **Pre-staged, mirrored images** — supply chain stays available and pinned, even offline.
+- **AWX job-level reporting** — every run is visible and auditable.
+
+## Outcome
+
+New edge sites are provisioned from code with consistent results, manual build steps are
+removed wherever logic allows, and the whole fleet is reproducible — an IaC-first build
+instead of a runbook.
+
+## Future improvements
+
+Tighten the loop from commit to provisioned site, and fold image-mirror freshness into the
+same pipeline so air-gapped caches are never silently stale.
@@ -0,0 +1,51 @@
+---
+title: "Self-Hosted AI & Homelab Platform"
+outcome: "A production-grade homelab — GitOps from bare metal to local AI, and the platform that serves this very site."
+summary: "Proxmox with PCIe passthrough under Talos and OpenShift clusters, all driven by ArgoCD GitOps: local LLM inference, split-horizon DNS, 2FA/SSO VPN, full observability and NAS-backed backups."
+role: "Owner / Operator"
+period: "Ongoing"
+stack: ["Talos", "OpenShift", "ArgoCD", "Proxmox", "Local LLM", "Cloudflare Tunnel"]
+featured: true
+order: 15
+diagram: "homelab"
+---
+
+## Problem
+
+The best way to stay sharp on platform engineering is to run a real platform — one with the
+same rigour as production, where the only person on call is you. The goal: a homelab that's a
+genuine proving ground for Kubernetes, GPUs, AI and security, not a pile of containers.
+
+## Constraints
+
+- **Run it like production** — GitOps, backups, observability, no snowflake config.
+- **Secure by default** — nothing exposed that doesn't need to be.
+- **Reproducible** — rebuild a node from code, not from memory.
+
+## Design
+
+Proxmox provides the hypervisor layer with **PCIe passthrough** (GPU and storage) into
+single-node **Talos** and **OpenShift** clusters. Everything is **ArgoCD GitOps** — the
+cluster state lives in git and reconciles itself. On top: **local LLM inference** on a
+Blackwell-class GPU, **split-horizon DNS** via Pi-hole, a VPN with **2FA/SSO**, and a
+**Prometheus / Grafana** observability stack. ZFS handles storage tiering; restic ships
+**NAS-backed backups**. Public services reach the internet through a **Cloudflare Tunnel** —
+which is exactly how this site is served.
+
+## Security & reliability decisions
+
+- **GitOps as the source of truth** — drift is reconciled, not chased.
+- **2FA / SSO and segmented access** — least privilege across the lab.
+- **Back up state, not just volumes** — restores are drilled, not hoped for.
+- **Outbound-only public exposure** — a tunnel, not an open port.
+
+## Outcome
+
+A homelab that behaves like a platform: rebuildable from code, observable, backed up, and
+secure enough to host a public site on. It's where new patterns get proven before they go
+anywhere near real infrastructure — and it's running right now, under this page.
+
+## Future improvements
+
+Continue migrating workloads to a unified GitOps story across clusters, and harden the
+edge-to-cluster path as more public services come online.