---
title: "Observe first, deny second"
date: 2026-07-14
summary: "Everyone writes network policy from the architecture diagram, and the diagram is always wrong. The only allowlist that survives contact with production is one written from the flows you actually watched - applied out-of-band, proven enforcing, and only then handed to GitOps."
tags: ["security", "networking", "cilium", "hubble", "kubernetes", "gitops"]
draft: false
hero: "/blog/observe-first-deny-second.webp"
heroAlt: "A dark reserve at night, instrument beams tracing animal paths before any fence exists"
---

Here's the uncomfortable default: when it's time to lock a namespace down, almost everyone opens the
architecture diagram and starts writing allow rules from it. This talks to that, that talks to the database,
done. The diagram is confident, tidy, and wrong. It omits the probe traffic, the DNS hop, the controller's
back-channel to the API server, the one call some library makes that nobody documented.

Policy written from a diagram *feels* like engineering. It's fiction with enforcement attached - and
enforcement doesn't care that the fiction was well-intentioned. It drops the flow anyway, in production,
at whatever hour the flow next happens.

I made the case for the default-deny floor itself in [the trust-nothing post](/blog/trust-nothing); this is
the other half: how you roll it across a live fleet, namespace by namespace, without an outage. The inversion is
simple. Don't fence first and see what starves. **Survey the reserve, then build the fence.**

<!-- DIAGRAM: the loop - Hubble observe -> write allowlist from evidence -> apply out-of-band -> verify enforcing + zero drops -> commit to git -->
![The observe -> deny -> verify -> commit loop](/diagrams/observe-first-deny-second.svg)

## Survey before you fence

A ranger doesn't fence a reserve from a map. They track the actual animals for a season - where they drink,
which paths they take at night, because the fence has to leave every real path open and close everything else.

Hubble is that season of tracking. Before a single policy exists, watch the namespace's real flows - every
connection, source and destination identity, port. That record *is* the allowlist. Not the docs, not your
memory of the architecture, not the diagram. Every rule you write should point at a flow you watched happen.
Start with the lowest-risk namespaces and leave the control plane for last: mis-fence a leaf app and one thing
breaks; mis-fence the plane that runs deploys and everything does.

## Build the fence where you can tear it down

Apply the policy trio (the default-deny plus its allows) **out-of-band** with `kubectl`, not through git.
This is deliberate, and it's the step people skip. The GitOps reconciler doesn't know the policy exists, so
it can't fight you over it, and rollback is one `kubectl delete`: instant, no commit, no sync wait. Push the
policy through git first and you've inverted your own safety: selfHeal means the reconciler restores whatever
you delete, in seconds, while you stand there believing you rolled back.

Out-of-band is a temporary state, not a destination. It's the fence held up with clamps while you check the
gates.

## Prove it bites, prove nothing bleeds

Two verifications, and both are mandatory because each one lies without the other.

First, prove the policy enforces at all. On Cilium 1.19 a policy can be **accepted but inert** - it passes a
server-side dry-run, sits in the cluster looking correct, and enforces nothing. A policy that looks live and
isn't is worse than no policy: it changes what you believe without changing what the network does. Read the
live object's status conditions and demand `Valid=True`, then watch a forbidden connection get dropped. "It
applied cleanly" and "it's enforcing" are different sentences.

Second, prove nothing legitimate is bleeding. Back to Hubble - zero drops on real flows, and the real consumer
path exercised end-to-end, not just a curl from your own shell.

Only when both hold does the policy go to git. The reconciler adopts it, out-of-band ends, and the rollback
story changes shape: from here, undo means `git revert`, because selfHeal now defends the policy as hard as
it would have fought your rollback.

The sternest test so far wasn't a rollout at all: the fleet's datapath was swapped out underneath the
policies - kube-proxy replaced by Cilium's eBPF kube-proxy-replacement on the last three clusters, one reboot
each, and afterwards every namespace's floor re-verified enforcing: 30/30, 32/32, 27/27 `Valid=True`, zero
legitimate drops. The fence held while the ground under it was replaced.

## The paths nobody draws

Three flows exist in every namespace and appear on no diagram. Omit any one and you pay:

- **DNS: allowed by label, never by IP.** Pin the resolver's ClusterIP into a rule and it never matches at
  all: the address is rewritten to a backend pod before policy judges the flow. Select `k8s-app: kube-dns`
  and it holds forever.
- **The kubelet's probes.** Health checks arrive from the **host** entity on *every* workload. Forget the
  allow and your pods go NotReady the moment the deny lands, the fence starving the animals it was meant
  to protect.
- **API-server egress.** Every controller and operator talks to the Kubernetes API constantly. A controller
  that "manages X" usually manages it *through* the API server, so this one allow often covers its whole
  lifecycle.

## Write enforcement from evidence, not assumption

The method generalises past network policy. Any control that turns assumption into enforcement - firewall,
RBAC, admission - is only as honest as the evidence it was written from. Observation first, out-of-band while
unproven, permanent only after it's earned it.

Policy written from observation is engineering. Policy written from a diagram is fiction, and production is
where fiction gets fact-checked.

*Rolled out across all six Talos clusters on Cilium 1.19 - every namespace surveyed with Hubble before its
deny landed, the control plane last, zero outages; the floor re-verified enforcing after this week's
kube-proxy->eBPF datapath swap.*
