---
title: "Init-gating GPU readiness on Kubernetes"
date: 2026-06-10
summary: "The single highest-leverage reliability fix for edge GPU workloads: never let an inference pod schedule before the GPU is actually ready."
tags: ["kubernetes", "gpu", "edge", "reliability"]
hero: "/blog/init-gating-gpu-readiness.webp"
heroAlt: "A GPU glows behind a sealed checkpoint gate while a waiting pod-orb is held in front; a readiness beacon shifts from red to green."

---

The most common way a GPU workload fails at the edge isn't the model, the driver, or the network.
It's timing. Kubernetes is eager — it will happily schedule your inference pod the moment a node
reports `Ready`, which is often *before* the NVIDIA device plugin has advertised `nvidia.com/gpu`.
The pod starts, can't see a GPU, crash-loops — and now your rollout is poisoned across the fleet,
on boxes nobody is standing next to.

Node-`Ready` answers the wrong question. It says the kubelet is up. It says nothing about whether
the one piece of hardware your workload exists to use is actually there yet. So stop trusting it:
make GPU readiness explicit, and gate on that.

## Gate the schedule, then gate the start

The first gate is free — a resource request. A pod that *requests* a GPU won't schedule until the
device plugin advertises capacity:

```yaml
resources:
  limits:
    nvidia.com/gpu: 1
```

That handles the common case. But on a single-GPU edge node recovering from a power cut, there's a
window where the plugin has advertised the device and the driver is still finding its feet — and
you don't want an expensive model load to be the thing that discovers it. So the second gate is an
init container that blocks until the device is demonstrably real, and fails loudly if it never is:

```bash
#!/usr/bin/env bash
set -euo pipefail
# Block until the GPU is visible AND healthy, or fail loudly after a bound.
for i in $(seq 1 30); do
  if nvidia-smi -L | grep -q '^GPU 0'; then
    echo "GPU ready"; exit 0
  fi
  echo "waiting for GPU ($i/30)"; sleep 5
done
echo "GPU never became ready" >&2
exit 1
```

Two gates, two failure modes closed: the scheduler can't place the pod before capacity exists, and
the workload can't start before the hardware answers. Note the bound — an init gate that waits
forever isn't a gate, it's a hang. Two and a half minutes, then fail loud and let the platform
retry. Fail-closed, never fail-quiet.

## Why this is the win

Once readiness is gated, the entire class of "pod started before the GPU" failures disappears —
and it disappears *the same way on every node*. That consistency is the real prize at the edge.
A fix that requires a human to notice, shell in, and nurse a bad rollout doesn't scale past the
first dozen sites; a gate that makes every node converge identically after every reboot does.

## The principle

At the edge, **design the dependency — don't hope for it**. Anything your workload cannot run
without deserves an explicit, bounded, fail-loud gate between it and the scheduler's optimism.
The GPU is just the first dependency worth naming; egress paths and model artifacts are next,
and they want the same treatment.
