---
title: "Six days of nothing, and nothing noticed"
date: 2026-08-17
summary: "A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed."
tags: ["observability", "monitoring", "reliability", "alerting", "homelab"]
draft: false
hero: "/blog/six-days-of-silence.webp"
heroAlt: "A wall of uniformly lit all-clear indicator lamps above a floor pipeline that has visibly run dry"
---

It was found by accident. Someone asked how the downloads were going, and the answer was that they
had not gone anywhere in six days. The tunnel that carries them had failed and the process
responsible had kept running - alive, responsive, and completely useless.

Nothing alerted, because nothing was watching the tunnel. The container was up. The platform had no
reason to think otherwise.

<!-- DIAGRAM: a container running with a healthy process while its tunnel is down, no probe covering the tunnel, and the six-day gap between failure and discovery -->
![Diagram of a running container with a healthy process beside its failed tunnel, with no probe covering the tunnel and a six-day gap before discovery](/diagrams/six-days-of-silence.svg)

## Running was never the job

The gap between "the process is running" and "the process is doing its job" is where invisible
outages live. A tunnel client with a dead tunnel is still a healthy process. It answers, it holds
its port, it logs. It simply is not carrying anything, and no default check in the stack has an
opinion about that.

Worse, the failure was self-obscuring. Its internal retry backoff kept growing, so the logs got
quieter over time rather than louder. By day six, the system was producing almost no evidence that
anything was wrong.

## Health lives in the flow

What closed the gap was probing the capability rather than the container: the tunnel's own health
check, which only passes when traffic genuinely egresses through it. That converts a silent failure
into a restart and, eventually, an alert - a thing the platform can see and act on.

The general rule: for anything whose value is a *flow*, health means the flow works. Ask whether
data recently moved, not whether the mover is alive.

## The question that found it

Health checks should assert the outcome the component exists to produce. A running process proves
almost nothing about a system whose job is to move something, and a failure nobody is watching for
will always be discovered the way this one was - by a person, on the day they happen to ask.

*Live in the lab: the tunnel's health check gates the container now - nothing reports healthy
unless traffic egresses through it.*
