blog: publish three posts — observe-first-deny-second, snat-ate-my-source, no-ssh
Heroes de-watermarked via the calibrated Jacobi diffuse (dewatermark.mjs now parameterised src/out), diagrams authored in the house light-card style.
This commit is contained in:
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: "My servers don't have SSH, and that's the feature"
|
||||
date: 2026-07-14
|
||||
summary: "Every box in this fleet runs an OS with no shell, no package manager, and no SSH daemon — the entire machine is an API with a declarative config. It sounds like giving up control. It's the opposite: you can't drift what you can't touch."
|
||||
tags: ["talos", "kubernetes", "immutable-infrastructure", "security", "homelab"]
|
||||
draft: false
|
||||
hero: "/blog/no-ssh.webp"
|
||||
heroAlt: "A sealed obsidian machine with a single structured API port of light"
|
||||
---
|
||||
|
||||
The first thing everyone asks about Talos Linux: how do you get in? You don't. There is no SSH daemon to
|
||||
connect to, no shell waiting behind it, no package manager if you got there, and no login even at the
|
||||
physical console. The operating system boots, runs Kubernetes, and answers exactly one thing: a mutual-TLS
|
||||
gRPC API. Six machines in this fleet, and not one of them is a place I can *visit*.
|
||||
|
||||
That sounds like a limitation you'd tolerate for the security. It took about a week of running it to
|
||||
realise it's the point.
|
||||
|
||||
## A server you can't visit is a server you can't drift
|
||||
|
||||
Every hand-run command on a traditional box is an unrecorded change: a config edited to test something, a
|
||||
package installed during an incident, a daemon restarted with a flag nobody wrote down. None of it is in
|
||||
git. All of it is load-bearing by next month. Configuration drift isn't a failure of discipline — it's the
|
||||
*inevitable* product of machines that accept hands.
|
||||
|
||||
Talos removes the hands. The entire machine — disks, network interfaces pinned by MAC, kernel modules,
|
||||
registry mirrors, even the manifests the cluster boots with — is one declarative document, applied through
|
||||
the API, versioned in git. There's no side door through which an undocumented change can arrive, because
|
||||
there's no door. The machine config isn't documentation *of* the machine. It **is** the machine.
|
||||
|
||||
<!-- DIAGRAM: left — a traditional server as a house with many doors (ssh, console, package manager), arrows of drift entering; right — a sealed Talos block with one structured API port, a signed config document flowing in -->
|
||||
|
||||
|
||||
## Operating through the keyhole
|
||||
|
||||
Day-two work changes shape. Logs, service status, process lists, even packet capture arrive through typed
|
||||
API calls, not a terminal session — and every mutation is an *apply*: edit the document, run a dry-run diff
|
||||
that says precisely what will change and whether it costs a reboot, then commit it. The dry-run gate is the
|
||||
quiet superpower — on a machine you can't shell into, "what exactly will this do?" stops being a guess and
|
||||
becomes an answer the API owes you *before* anything moves.
|
||||
|
||||
This week put that shape under load. Three of the six nodes had their network datapath swapped — kube-proxy
|
||||
out, Cilium's eBPF replacement in — and their registry credentials rotated, one reboot each, entirely
|
||||
through the API. Nobody logged into anything, because there is nothing to log into. The credentials never
|
||||
touched a live machine either: they're substituted into the document at render time from an encrypted file
|
||||
and arrive as configuration, not keystrokes — there is no terminal to mistype them into.
|
||||
|
||||
Two habits from that work show what keyhole operation feels like. After an apply that costs a reboot, the
|
||||
API answers again *before* the machine has cycled — the door reopening proves nothing. So you read
|
||||
`/proc/uptime` through the API and demand a number seconds old; the check has caught a node still showing
|
||||
413,223 seconds of uptime with the apply long returned. You don't trust the door. You read the building's
|
||||
own clock through the keyhole. And before calling a node good, you have it pull a container image through
|
||||
that same API — the machine exercises its new registry credential end-to-end while you watch from outside.
|
||||
|
||||
When something's genuinely wrong, the recovery isn't archaeology on a mutated filesystem — it's
|
||||
reconciliation: re-apply the known-good document, or in the worst case rebuild the node from it in minutes.
|
||||
The machine has no state worth rescuing *because nothing was ever hand-placed on it*. Pets die of unknown
|
||||
illnesses; documents get reprinted.
|
||||
|
||||
## The honest trade
|
||||
|
||||
The console is gone as a *control* surface, not as a *truth* surface — and that distinction still bites. A
|
||||
node once dropped off the network in a way that looked, from every remote signal, like a total lockup. The
|
||||
physical screen told a different story: the box was alive and healthy, only its network path had wedged.
|
||||
The lesson isn't "you need a shell" — a shell would have shown the same thing more slowly. It's that
|
||||
firsthand evidence still outranks remote inference, and an API-only fleet needs its operators to remember
|
||||
the difference on the day it matters. The same lesson came back this week when a node went dark at layer 2:
|
||||
the fix was a screwdriver, not a session — no SSH daemon answers on a dead NIC either.
|
||||
|
||||
## The principle
|
||||
|
||||
Control isn't the ability to touch a machine — it's the guarantee that the machine matches a document you
|
||||
trust. Every interactive door a server offers is a place where reality and the record can quietly diverge,
|
||||
and everything that makes fleets miserable lives in that gap. Seal the doors, apply the document, keep the
|
||||
diff. The machine you can't log into is the only one whose state you truly know.
|
||||
|
||||
*Live across a six-node Talos fleet: machine configs generated and versioned in git, applied over mTLS with
|
||||
dry-run gates, nodes rebuilt from documents when hardware moves. Three of the six had their network datapath
|
||||
swapped and their registry credentials rotated this week — one reboot each, entirely through the API. No SSH
|
||||
key to any of them exists, because there is nothing for it to open.*
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: "Observe first, deny second"
|
||||
date: 2026-07-14
|
||||
summary: "Everyone writes network policy from the architecture diagram, and the diagram is always wrong. The only allowlist that survives contact with production is one written from the flows you actually watched — applied out-of-band, proven enforcing, and only then handed to GitOps."
|
||||
tags: ["security", "networking", "cilium", "hubble", "kubernetes", "gitops"]
|
||||
draft: false
|
||||
hero: "/blog/observe-first-deny-second.webp"
|
||||
heroAlt: "A dark reserve at night, instrument beams tracing animal paths before any fence exists"
|
||||
---
|
||||
|
||||
Here's the uncomfortable default: when it's time to lock a namespace down, almost everyone opens the
|
||||
architecture diagram and starts writing allow rules from it. This talks to that, that talks to the database,
|
||||
done. The diagram is confident, tidy — and wrong. It omits the probe traffic, the DNS hop, the controller's
|
||||
back-channel to the API server, the one call some library makes that nobody documented.
|
||||
|
||||
Policy written from a diagram *feels* like engineering. It's fiction with enforcement attached — and
|
||||
enforcement doesn't care that the fiction was well-intentioned. It drops the flow anyway, in production,
|
||||
at whatever hour the flow next happens.
|
||||
|
||||
I made the case for the default-deny floor itself in [the trust-nothing post](/blog/trust-nothing); this is
|
||||
the other half: how you roll it across a live fleet, namespace by namespace, without an outage. The inversion is
|
||||
simple. Don't fence first and see what starves. **Survey the reserve, then build the fence.**
|
||||
|
||||
<!-- DIAGRAM: the loop — Hubble observe → write allowlist from evidence → apply out-of-band → verify enforcing + zero drops → commit to git -->
|
||||
|
||||
|
||||
## Survey before you fence
|
||||
|
||||
A ranger doesn't fence a reserve from a map. They track the actual animals for a season — where they drink,
|
||||
which paths they take at night — because the fence has to leave every real path open and close everything else.
|
||||
|
||||
Hubble is that season of tracking. Before a single policy exists, watch the namespace's real flows — every
|
||||
connection, source and destination identity, port. That record *is* the allowlist. Not the docs, not your
|
||||
memory of the architecture, not the diagram. Every rule you write should point at a flow you watched happen.
|
||||
Start with the lowest-risk namespaces and leave the control plane for last: mis-fence a leaf app and one thing
|
||||
breaks; mis-fence the plane that runs deploys and everything does.
|
||||
|
||||
## Build the fence where you can tear it down
|
||||
|
||||
Apply the policy trio — the default-deny plus its allows — **out-of-band** with `kubectl`, not through git.
|
||||
This is deliberate, and it's the step people skip. The GitOps reconciler doesn't know the policy exists, so
|
||||
it can't fight you over it, and rollback is one `kubectl delete` — instant, no commit, no sync wait. Push the
|
||||
policy through git first and you've inverted your own safety: selfHeal means the reconciler restores whatever
|
||||
you delete, in seconds, while you stand there believing you rolled back.
|
||||
|
||||
Out-of-band is a temporary state, not a destination. It's the fence held up with clamps while you check the
|
||||
gates.
|
||||
|
||||
## Prove it bites, prove nothing bleeds
|
||||
|
||||
Two verifications, and both are mandatory because each one lies without the other.
|
||||
|
||||
First, prove the policy enforces at all. On Cilium 1.19 a policy can be **accepted but inert** — it passes a
|
||||
server-side dry-run, sits in the cluster looking correct, and enforces nothing. A policy that looks live and
|
||||
isn't is worse than no policy: it changes what you believe without changing what the network does. Read the
|
||||
live object's status conditions and demand `Valid=True`, then watch a forbidden connection get dropped. "It
|
||||
applied cleanly" and "it's enforcing" are different sentences.
|
||||
|
||||
Second, prove nothing legitimate is bleeding. Back to Hubble — zero drops on real flows, and the real consumer
|
||||
path exercised end-to-end, not just a curl from your own shell.
|
||||
|
||||
Only when both hold does the policy go to git. The reconciler adopts it, out-of-band ends, and the rollback
|
||||
story changes shape: from here, undo means `git revert` — because selfHeal now defends the policy as hard as
|
||||
it would have fought your rollback.
|
||||
|
||||
The sternest test so far wasn't a rollout at all: the fleet's datapath was swapped out underneath the
|
||||
policies — kube-proxy replaced by Cilium's eBPF kube-proxy-replacement on the last three clusters, one reboot
|
||||
each — and afterwards every namespace's floor re-verified enforcing: 30/30, 32/32, 27/27 `Valid=True`, zero
|
||||
legitimate drops. The fence held while the ground under it was replaced.
|
||||
|
||||
## The paths nobody draws
|
||||
|
||||
Three flows exist in every namespace and appear on no diagram. Omit any one and you pay:
|
||||
|
||||
- **DNS — allowed by label, never by IP.** Pin the resolver's ClusterIP into a rule and it never matches at
|
||||
all — the address is rewritten to a backend pod before policy judges the flow. Select `k8s-app: kube-dns`
|
||||
and it holds forever.
|
||||
- **The kubelet's probes.** Health checks arrive from the **host** entity on *every* workload. Forget the
|
||||
allow and your pods go NotReady the moment the deny lands — the fence starving the animals it was meant
|
||||
to protect.
|
||||
- **API-server egress.** Every controller and operator talks to the Kubernetes API constantly. A controller
|
||||
that "manages X" usually manages it *through* the API server, so this one allow often covers its whole
|
||||
lifecycle.
|
||||
|
||||
## The principle
|
||||
|
||||
The method generalises past network policy. Any control that turns assumption into enforcement — firewall,
|
||||
RBAC, admission — is only as honest as the evidence it was written from. Observation first, out-of-band while
|
||||
unproven, permanent only after it's earned it.
|
||||
|
||||
Policy written from observation is engineering. Policy written from a diagram is fiction — and production is
|
||||
where fiction gets fact-checked.
|
||||
|
||||
*Rolled out across all six Talos clusters on Cilium 1.19 — every namespace surveyed with Hubble before its
|
||||
deny landed, the control plane last, zero outages; the floor re-verified enforcing after this week's
|
||||
kube-proxy→eBPF datapath swap.*
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "SNAT ate my source IP"
|
||||
date: 2026-07-14
|
||||
summary: "A LoadBalancer service with the default traffic policy rewrites every incoming packet's source to the node's own address — so by the time a network policy sees it, the real client is gone. You cannot allowlist a sender the network has already erased."
|
||||
tags: ["security", "networking", "cilium", "kubernetes", "load-balancing"]
|
||||
draft: false
|
||||
hero: "/blog/snat-ate-my-source.webp"
|
||||
heroAlt: "A glowing envelope passing through a dark sorting machine that stamps over its return address"
|
||||
---
|
||||
|
||||
Here's the uncomfortable default: give a Kubernetes service a LoadBalancer IP, leave
|
||||
`externalTrafficPolicy` at its default of `Cluster`, and **every packet that arrives gets its source
|
||||
address rewritten to the node's own**. Not by an attacker. By the load balancer, on purpose, as
|
||||
routine plumbing.
|
||||
|
||||
I found out the way you always find out — by writing a network policy for it. LAN clients live on
|
||||
`10.0.11.0/24`, so I wrote a `fromCIDR 10.0.11.0/24` allow. Applied cleanly, showed `Valid=True`,
|
||||
dropped every real user. Hubble told me why: the traffic wasn't arriving from the LAN at all. Cilium
|
||||
tagged the source `world` — the identity of *anywhere*, the one you can't restrict without
|
||||
restricting everything.
|
||||
|
||||
It's a letter that reaches you carrying the sorting office's return address instead of the sender's.
|
||||
The sorting office isn't lying to you; it's how the machinery forwards mail. But you can't write a
|
||||
"letters from Alice only" rule when every envelope on your doormat says it came from the depot.
|
||||
|
||||
<!-- DIAGRAM: client → LB VIP → SNAT at the node (source rewritten) → policy sees `world`, vs ETP:Local preserving the client address -->
|
||||
|
||||
|
||||
## Why the depot stamps over the sender
|
||||
|
||||
The rewrite has a reason. With `externalTrafficPolicy: Cluster`, any node can accept traffic for the
|
||||
service and forward it to a backend pod on a *different* node. For the reply to route back through
|
||||
the node that forwarded it, that node SNATs the connection to itself. Balanced spreading, bought by
|
||||
destroying the source.
|
||||
|
||||
The consequence lands exactly where you can't see it coming: **policies bind to what arrives, not to
|
||||
what was sent.** The client's packet left home with an honest return address; the depot stamped over
|
||||
it in transit; your policy — evaluated after the rewrite — never meets the client at all. There is no
|
||||
rule you can write for an identity the network erased one hop earlier.
|
||||
|
||||
## Preserve the sender, then commit it
|
||||
|
||||
The fix is one line: `externalTrafficPolicy: Local`. Only nodes running a backend pod accept
|
||||
the traffic, no cross-node forwarding happens, no SNAT is needed — the client's real address survives
|
||||
to policy evaluation. Now `fromCIDR 10.0.11.0/24` matches LAN clients, and a cross-cluster peer shows
|
||||
up as its node IP — the peer *node*, not the peer pod, because separate clusters masquerade pod
|
||||
traffic on the way out — pin-downable with a `/32`. That's how the LLM gateway's ingress rule got to
|
||||
be a single line: one legitimate consumer, the agent cluster next door, exactly one `/32` allowed in.
|
||||
|
||||
Here's the one that costs a 2am. I made that change with `kubectl patch` — quick, out-of-band, worked
|
||||
immediately. The service was GitOps-managed with self-heal on. Within seconds ArgoCD noticed the live
|
||||
object differed from git and put it back the way the repo said, which re-enabled the SNAT, which
|
||||
re-broke the policy, which took DNS down with it. **The change must be committed**, or the platform
|
||||
will politely undo your fix while you sleep.
|
||||
|
||||
One caveat that keeps it honest: some paths still rewrite. A client arriving through a
|
||||
tailnet subnet router gets masqueraded at the routing node even with `Local` set — it lands as
|
||||
`world` regardless. Some envelopes pass through a second depot you don't control.
|
||||
|
||||
## When the depot closes, every envelope changes
|
||||
|
||||
This week the fleet finished swapping kube-proxy for Cilium's eBPF replacement. Same services, same
|
||||
addresses, same policies — and a traefik that had answered LAN probes for months went dark, under a
|
||||
policy nobody touched.
|
||||
|
||||
It was never supposed to answer. That traefik fronts zero routes, and the tightest floor guards it —
|
||||
a policy admitting `host`, because kubelet probes need it, and nothing else. But kube-proxy's SNAT
|
||||
had been dressing LAN probes as `host` — a different depot, a different stamp, the same disease. The
|
||||
replacement closed the depot. Every envelope arrived carrying its real return address, the probes
|
||||
landed as `world`, and the policy dropped them — exactly as written. Nothing broke. The policy
|
||||
finally saw the truth, and the truth matched the original intent: answer no one.
|
||||
|
||||
But if you didn't know *why* it used to answer, this is the morning you file an outage ticket and
|
||||
"fix" a policy that was never wrong. The rewriting machinery is itself a moving part. Change the
|
||||
plumbing and every identity changes with it — which is why only observed flows stay true, and
|
||||
remembered ones quietly expire.
|
||||
|
||||
## Read the envelope before you write the rule
|
||||
|
||||
The real lesson isn't the one-line fix. It's that I wrote a policy for the traffic I *imagined* —
|
||||
clients on the LAN, arriving as themselves — instead of the traffic that *arrived*. The packet's
|
||||
story gets rewritten at every hop: SNAT here, DNAT there, a masquerade at a routing boundary. The
|
||||
sender's truth and the receiver's truth are different documents.
|
||||
|
||||
So the discipline, before any allow rule exists: **observe first.** Open Hubble, watch the real
|
||||
flows, and note the identity the traffic actually carries when it reaches the endpoint — `world`,
|
||||
`host`, `cluster`, a pod label, a CIDR. Then write the rule for *that*. A policy written from the
|
||||
architecture diagram is a guess; a policy written from observed flows is a fact.
|
||||
|
||||
## The principle
|
||||
|
||||
A network policy is a doorman checking return addresses, and the postal system rewrites return
|
||||
addresses as a matter of course. You don't secure what was sent — nobody ever sees what was sent.
|
||||
You secure what arrives. Go and look at the envelope first.
|
||||
|
||||
*Bitten and fixed on a six-cluster Talos fleet running Cilium 1.19 — kube-proxy replaced by the eBPF
|
||||
datapath fleet-wide, MetalLB retired, and a standing rule: Hubble before policy, every time.*
|
||||
Reference in New Issue
Block a user