---
title: "The cert that couldn't see its own proof"
date: 2026-07-03
summary: "Let's Encrypt DNS-01 posted its proof to the public internet — and then checked for it through the LAN's split-horizon DNS, which will never carry it. The fix is one line: make the self-check look where the CA looks, not where you live."
tags: ["tls", "dns", "cert-manager", "lets-encrypt", "split-horizon"]
hero: "/blog/cert-couldnt-see-its-proof.webp?v=1"
heroAlt: "A sealed structure split by a translucent wall: a glowing proof-token sits in the open on the outside where a distant beam finds it, while an identical vantage on the inside looks toward the same spot and sees nothing."
draft: false
---


Here's the uncomfortable default: cert-manager checks its own homework **through whatever DNS the cluster hands it**. On a normal network that's fine — the cluster's resolvers see the same internet everyone else does. On a split-horizon network, they don't. Your internal DNS is authoritative for the domain, answers first, and knows nothing about the record you just published to the world.

So the DNS-01 challenge does something quietly absurd. cert-manager writes the `_acme-challenge` TXT record to Cloudflare — and it lands. `dig @1.1.1.1` returns the token. Let's Encrypt could validate it right now. But before asking the CA to look, cert-manager runs a *self-check*: it resolves the record itself, through cluster DNS, through the Pi-hole that owns `bztmon.org` internally and has no such TXT record. Nothing comes back. So it waits, and checks again, and loops on `"not yet propagated"` — forever.

It's posting a letter into the public postbox, then walking back inside to check the hallway mail slot for it. The letter is out there. The postman can see it. You're looking in the one box it will never reach — and concluding it hasn't been sent.

<!-- DIAGRAM: split-horizon DNS-01 — cert-manager writes the TXT to Cloudflare (public path, record live), Let's Encrypt validates via public DNS, but the self-check arrow loops through the internal Pi-hole which returns nothing; the fix arrow reroutes the self-check to 1.1.1.1/8.8.8.8 -->
![The self-check resolving through the wrong horizon while the CA sees the record](/diagrams/cert-couldnt-see-its-proof.svg?v=1)

## Look where the examiner looks

The fix is one Helm value, and it encodes the whole lesson:

```yaml
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
dns01RecursiveNameserversOnly: true
```

That points the self-check at public resolvers — the same vantage Let's Encrypt validates from — instead of the cluster's own view of the world. The moment cert-manager checks the postbox the postman actually collects from, the challenge that had been pending for an age clears in seconds. No record changed. No DNS changed. Only the *observer* moved.

One footnote if you run default-deny egress: the check now leaves the cluster on port 53, so `world:53` has to be allowed for the cert-manager pods. Here it already was, courtesy of the same policy that lets cloudflared out.

## Two proofs, one postbox

The same challenge has a second way to hang, and it looks identical from the outside. Request a certificate for both the wildcard `*.bztmon.org` *and* the apex `bztmon.org`, and Let's Encrypt opens two authorisations — both of which validate at the **same** record name, `_acme-challenge.bztmon.org`, with *different* tokens. cert-manager's Cloudflare solver dutifully writes one, then the other solver overwrites it, then the first writes it back. Two letters fighting over one envelope, each self-check finding the other's contents. Both loop on `"not yet propagated"` — and neither is a propagation problem.

The fix is to stop asking for both: request the wildcard only, and drop the apex SAN unless you serve the bare domain. This one ate a real evening on robin before the pattern was obvious.

## A permission failure in a propagation costume

The third trap doesn't even mention permissions. A Cloudflare API token with `Zone:DNS:Edit` alone *looks* sufficient — it can edit records, what more could a DNS solver want? But cert-manager first has to enumerate zones to find the zone ID, and that needs **`Zone:Zone:Read`**. Without it, the failure surfaces not as `403` but as — you guessed it — `"not yet propagated"`.

Add Zone:Read to the token at Cloudflare and it unblocks instantly. The token string doesn't change, so no secret rotates; the fix never touches the cluster at all.

Three different root causes. One identical symptom. That's the real menace of this error message: it names the *last observation* ("I can't see the record"), not the reason.

## The principle

A self-check is only worth what its vantage point is worth. If your verifier stands outside — a CA on the public internet, a monitor beyond the LAN, an auditor reading the published copy — then a check that observes from *inside* isn't a weaker version of the truth. It's a different truth, and it can disagree indefinitely.

So when a system checks its own work before presenting it, ask one question first: does it look from where the judge will look? Check the postbox the postman collects from — not the mail slot in your own hallway.

*Bit the media-gateway wildcard on elfastc, 2026-07-02 — cert-manager v1.19.5 issuing `*.bztmon.org` via Cloudflare DNS-01, on a fleet where the internal Pi-hole owns the zone and public resolvers hold the proof.*
