---
title: "SNAT ate my source IP"
date: 2026-07-14
summary: "A LoadBalancer service with the default traffic policy rewrites every incoming packet's source to the node's own address, so by the time a network policy sees it, the real client is gone. You cannot allowlist a sender the network has already erased."
tags: ["security", "networking", "cilium", "kubernetes", "load-balancing"]
draft: false
hero: "/blog/snat-ate-my-source.webp"
heroAlt: "A glowing envelope passing through a dark sorting machine that stamps over its return address"
---

Here's the uncomfortable default: give a Kubernetes service a LoadBalancer IP, leave
`externalTrafficPolicy` at its default of `Cluster`, and **every packet that arrives gets its source
address rewritten to the node's own**. Not by an attacker. By the load balancer, on purpose, as
routine plumbing.

I found out the way you always find out - by writing a network policy for it. LAN clients live on
`10.0.11.0/24`, so I wrote a `fromCIDR 10.0.11.0/24` allow. Applied cleanly, showed `Valid=True`,
dropped every real user. Hubble told me why: the traffic wasn't arriving from the LAN at all. Cilium
tagged the source `world` - the identity of *anywhere*, the one you can't restrict without
restricting everything.

It's a letter that reaches you carrying the sorting office's return address instead of the sender's.
The sorting office isn't lying to you; it's how the machinery forwards mail. But you can't write a
"letters from Alice only" rule when every envelope on your doormat says it came from the depot.

<!-- DIAGRAM: client -> LB VIP -> SNAT at the node (source rewritten) -> policy sees `world`, vs ETP:Local preserving the client address -->
![SNAT rewriting the client source before policy evaluation, and ETP:Local preserving it](/diagrams/snat-ate-my-source.svg)

## Why the depot stamps over the sender

The rewrite has a reason. With `externalTrafficPolicy: Cluster`, any node can accept traffic for the
service and forward it to a backend pod on a *different* node. For the reply to route back through
the node that forwarded it, that node SNATs the connection to itself. Balanced spreading, bought by
destroying the source.

The consequence lands exactly where you can't see it coming: **policies bind to what arrives, not to
what was sent.** The client's packet left home with an honest return address; the depot stamped over
it in transit; your policy - evaluated after the rewrite - never meets the client at all. There is no
rule you can write for an identity the network erased one hop earlier.

## Preserve the sender, then commit it

The fix is one line: `externalTrafficPolicy: Local`. Only nodes running a backend pod accept
the traffic, no cross-node forwarding happens, no SNAT is needed, and the client's real address survives
to policy evaluation. Now `fromCIDR 10.0.11.0/24` matches LAN clients, and a cross-cluster peer shows
up as its node IP (the peer *node*, not the peer pod, because separate clusters masquerade pod
traffic on the way out), pin-downable with a `/32`. That's how the LLM gateway's ingress rule got to
be a single line: one legitimate consumer, the agent cluster next door, exactly one `/32` allowed in.

Here's the one that costs a 2am. I made that change with `kubectl patch` - quick, out-of-band, worked
immediately. The service was GitOps-managed with self-heal on. Within seconds Argo CD noticed the live
object differed from git and put it back the way the repo said, which re-enabled the SNAT, which
re-broke the policy, which took DNS down with it. **The change must be committed**, or the platform
will politely undo your fix while you sleep.

One caveat that keeps it honest: some paths still rewrite. A client arriving through a
tailnet subnet router gets masqueraded at the routing node even with `Local` set - it lands as
`world` regardless. Some envelopes pass through a second depot you don't control.

## When the depot closes, every envelope changes

This week the fleet finished swapping kube-proxy for Cilium's eBPF replacement. Same services, same
addresses, same policies, and a traefik that had answered LAN probes for months went dark, under a
policy nobody touched.

It was never supposed to answer. That traefik fronts zero routes, and the tightest floor guards it: a policy admitting `host`, because kubelet probes need it, and nothing else. But kube-proxy's SNAT
had been dressing LAN probes as `host`: a different depot, a different stamp, the same disease. The
replacement closed the depot. Every envelope arrived carrying its real return address, the probes
landed as `world`, and the policy dropped them, exactly as written. Nothing broke. The policy
finally saw the truth, and the truth matched the original intent: answer no one.

But if you didn't know *why* it used to answer, this is the morning you file an outage ticket and
"fix" a policy that was never wrong. The rewriting machinery is itself a moving part. Change the
plumbing and every identity changes with it, which is why only observed flows stay true, and
remembered ones quietly expire.

## Read the envelope before you write the rule

The real lesson isn't the one-line fix. It's that I wrote a policy for the traffic I *imagined* - clients on the LAN, arriving as themselves - instead of the traffic that *arrived*. The packet's
story gets rewritten at every hop: SNAT here, DNAT there, a masquerade at a routing boundary. The
sender's truth and the receiver's truth are different documents.

So the discipline, before any allow rule exists: **observe first.** Open Hubble, watch the real
flows, and note the identity the traffic actually carries when it reaches the endpoint - `world`,
`host`, `cluster`, a pod label, a CIDR. Then write the rule for *that*. A policy written from the
architecture diagram is a guess; a policy written from observed flows is a fact.

## Secure what arrives, not what was sent

A network policy is a doorman checking return addresses, and the postal system rewrites return
addresses as a matter of course. You don't secure what was sent - nobody ever sees what was sent.
You secure what arrives. Go and look at the envelope first.

*Bitten and fixed on a six-cluster Talos fleet running Cilium 1.19 - kube-proxy replaced by the eBPF
datapath fleet-wide, MetalLB retired, and a standing rule: Hubble before policy, every time.*
