blog: give each post its own closing heading instead of sixteen sections called 'The principle'
build-and-deploy / build (push) Failing after 13m5s
build-and-deploy / build (push) Failing after 13m5s
This commit is contained in:
@@ -52,7 +52,7 @@ Two traps ate real hours, and they're the same lesson wearing two hats:
|
|||||||
Both say the same thing: SSO has two channels - the one the user sees and the one the server uses, and they
|
Both say the same thing: SSO has two channels - the one the user sees and the one the server uses, and they
|
||||||
don't live on the same network.
|
don't live on the same network.
|
||||||
|
|
||||||
## The principle
|
## Design the concentration you just created
|
||||||
|
|
||||||
Centralising identity is a force multiplier, but it concentrates risk *by design*. So design the
|
Centralising identity is a force multiplier, but it concentrates risk *by design*. So design the
|
||||||
concentration: enforce the strong factor at the boundary, plan the failure mode before you need it, and
|
concentration: enforce the strong factor at the boundary, plan the failure mode before you need it, and
|
||||||
|
|||||||
@@ -66,7 +66,7 @@ The rule that falls out: a node on the only LAN **advertises routes but never ac
|
|||||||
network natively. And the recovery is itself a bastion lesson: you get back in *out-of-band*, through the
|
network natively. And the recovery is itself a bastion lesson: you get back in *out-of-band*, through the
|
||||||
hypervisor console. The control point must sit on infrastructure you can still reach when the network lies.
|
hypervisor console. The control point must sit on infrastructure you can still reach when the network lies.
|
||||||
|
|
||||||
## The principle
|
## One place holds the keys
|
||||||
|
|
||||||
A bastion isn't a product you install; it's a property you enforce: **the set of places your keys exist has
|
A bastion isn't a product you install; it's a property you enforce: **the set of places your keys exist has
|
||||||
exactly one member.** Any box can hold the role - a VM, a spare NUC, a cloud instance, and the role can move,
|
exactly one member.** Any box can hold the role - a VM, a spare NUC, a cloud instance, and the role can move,
|
||||||
|
|||||||
@@ -45,7 +45,7 @@ Add Zone:Read to the token at Cloudflare and it unblocks instantly. The token st
|
|||||||
|
|
||||||
Three different root causes. One identical symptom. That's the real menace of this error message: it names the *last observation* ("I can't see the record"), not the reason.
|
Three different root causes. One identical symptom. That's the real menace of this error message: it names the *last observation* ("I can't see the record"), not the reason.
|
||||||
|
|
||||||
## The principle
|
## A check is only worth its vantage point
|
||||||
|
|
||||||
A self-check is only worth what its vantage point is worth. If your verifier stands outside - a CA on the public internet, a monitor beyond the LAN, an auditor reading the published copy - then a check that observes from *inside* isn't a weaker version of the truth. It's a different truth, and it can disagree indefinitely.
|
A self-check is only worth what its vantage point is worth. If your verifier stands outside - a CA on the public internet, a monitor beyond the LAN, an auditor reading the published copy - then a check that observes from *inside* isn't a weaker version of the truth. It's a different truth, and it can disagree indefinitely.
|
||||||
|
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ Under both sits a default-deny network policy. The pod can reach the git server,
|
|||||||
|
|
||||||
Just as important are the doors that don't exist. A direct LAN bind and a separate mesh-VPN serving path were both sketched, and both cut - the one guarded door already serves the couch and the phone alike. Every additional entrance is an additional thing to prove, forever.
|
Just as important are the doors that don't exist. A direct LAN bind and a separate mesh-VPN serving path were both sketched, and both cut - the one guarded door already serves the couch and the phone alike. Every additional entrance is an additional thing to prove, forever.
|
||||||
|
|
||||||
## The principle
|
## Work backwards from a total compromise
|
||||||
|
|
||||||
Every new convenience surface - a browser IDE, a remote dashboard, a sync agent - should be designed backwards from one number: what a *total* compromise of it yields. Compute that before the surface exists, while the answer is still a design choice. Then engineer it down: clones instead of originals, an identity scoped to the surface and revocable without ceremony, doors layered so no single failure is enough. Only when the worst case is something you could live through do you get to enjoy the convenience, and then you get to enjoy it completely, from any couch you like.
|
Every new convenience surface - a browser IDE, a remote dashboard, a sync agent - should be designed backwards from one number: what a *total* compromise of it yields. Compute that before the surface exists, while the answer is still a design choice. Then engineer it down: clones instead of originals, an identity scoped to the surface and revocable without ceremony, doors layered so no single failure is enough. Only when the worst case is something you could live through do you get to enjoy the convenience, and then you get to enjoy it completely, from any couch you like.
|
||||||
|
|
||||||
|
|||||||
@@ -59,7 +59,7 @@ Once readiness is gated, the entire class of "pod started before the GPU" failur
|
|||||||
A fix that requires a human to notice, shell in, and nurse a bad rollout doesn't scale past the
|
A fix that requires a human to notice, shell in, and nurse a bad rollout doesn't scale past the
|
||||||
first dozen sites; a gate that makes every node converge identically after every reboot does.
|
first dozen sites; a gate that makes every node converge identically after every reboot does.
|
||||||
|
|
||||||
## The principle
|
## Design the dependency, don't hope for it
|
||||||
|
|
||||||
At the edge, **design the dependency - don't hope for it**. Anything your workload cannot run
|
At the edge, **design the dependency - don't hope for it**. Anything your workload cannot run
|
||||||
without deserves an explicit, bounded, fail-loud gate between it and the scheduler's optimism.
|
without deserves an explicit, bounded, fail-loud gate between it and the scheduler's optimism.
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ kubectl apply --server-side --force-conflicts --field-manager=talos -f cilium-re
|
|||||||
|
|
||||||
The second was musical chairs with the VIPs. Mid-swap on one cluster, a service without a pinned address grabbed another service's freed IP the instant the old announcer released it - the LLM backend ended up sitting on the ingress controller's address. The fix is an ordering rule: pins first. Land the address pins while the old announcer is still alive, confirm they hold, and only then retire it. Two commits, in that order, every time.
|
The second was musical chairs with the VIPs. Mid-swap on one cluster, a service without a pinned address grabbed another service's freed IP the instant the old announcer released it - the LLM backend ended up sitting on the ingress controller's address. The fix is an ordering rule: pins first. Land the address pins while the old announcer is still alive, confirm they hold, and only then retire it. Two commits, in that order, every time.
|
||||||
|
|
||||||
## The principle
|
## Count what you removed, not what you added
|
||||||
|
|
||||||
Consolidation is usually sold on the new tool's feature list. That's the wrong ledger. The value of landing on one engine wasn't what Cilium added - it was what the landing let us remove: a CNI, a service proxy, a VIP announcer, and on one node an ingress controller, each of which had been a version to pin, a feed to watch, an alert to tune, a way to be paged. Policy enforcement and flow observability never needed projects of their own; they came up with the same engine, side effects of the consolidation rather than line items on it.
|
Consolidation is usually sold on the new tool's feature list. That's the wrong ledger. The value of landing on one engine wasn't what Cilium added - it was what the landing let us remove: a CNI, a service proxy, a VIP announcer, and on one node an ingress controller, each of which had been a version to pin, a feed to watch, an alert to tune, a way to be paged. Policy enforcement and flow observability never needed projects of their own; they came up with the same engine, side effects of the consolidation rather than line items on it.
|
||||||
|
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ firsthand evidence still outranks remote inference, and an API-only fleet needs
|
|||||||
the difference on the day it matters. The same lesson came back this week when a node went dark at layer 2:
|
the difference on the day it matters. The same lesson came back this week when a node went dark at layer 2:
|
||||||
the fix was a screwdriver, not a session - no SSH daemon answers on a dead NIC either.
|
the fix was a screwdriver, not a session - no SSH daemon answers on a dead NIC either.
|
||||||
|
|
||||||
## The principle
|
## Control is the machine matching its document
|
||||||
|
|
||||||
Control isn't the ability to touch a machine - it's the guarantee that the machine matches a document you
|
Control isn't the ability to touch a machine - it's the guarantee that the machine matches a document you
|
||||||
trust. Every interactive door a server offers is a place where reality and the record can quietly diverge,
|
trust. Every interactive door a server offers is a place where reality and the record can quietly diverge,
|
||||||
|
|||||||
@@ -82,7 +82,7 @@ Three flows exist in every namespace and appear on no diagram. Omit any one and
|
|||||||
that "manages X" usually manages it *through* the API server, so this one allow often covers its whole
|
that "manages X" usually manages it *through* the API server, so this one allow often covers its whole
|
||||||
lifecycle.
|
lifecycle.
|
||||||
|
|
||||||
## The principle
|
## Write enforcement from evidence, not assumption
|
||||||
|
|
||||||
The method generalises past network policy. Any control that turns assumption into enforcement - firewall,
|
The method generalises past network policy. Any control that turns assumption into enforcement - firewall,
|
||||||
RBAC, admission - is only as honest as the evidence it was written from. Observation first, out-of-band while
|
RBAC, admission - is only as honest as the evidence it was written from. Observation first, out-of-band while
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ The browser IDE made this concrete. Its whole value is a terminal speaking over
|
|||||||
|
|
||||||
So the route sends every path on the host to the outpost, and the check was empirical: an unauthenticated request to the root, to the terminal WebSocket, to a static asset and to the health endpoint each came back as a redirect to the identity provider. All of them. On a human door, a single carve-out *is* the hole; carve-outs belong only on machine doors, where they get keys of their own. And behind the doorman, the IDE still keeps a lock of its own - the desk adds a layer; it doesn't replace the one on the office door.
|
So the route sends every path on the host to the outpost, and the check was empirical: an unauthenticated request to the root, to the terminal WebSocket, to a static asset and to the health endpoint each came back as a redirect to the identity provider. All of them. On a human door, a single carve-out *is* the hole; carve-outs belong only on machine doors, where they get keys of their own. And behind the doorman, the IDE still keeps a lock of its own - the desk adds a layer; it doesn't replace the one on the office door.
|
||||||
|
|
||||||
## The principle
|
## Slice auth by who arrives
|
||||||
|
|
||||||
Slice authentication by who arrives, not by what serves the request. Humans are one audience - interactive, browser-borne, phishable, and they get a doorman: one strong flow, every human door in the estate behind the same desk, so door N+1 inherits the whole policy for the cost of a nameplate. Machines are the other audience - headless, scoped, incapable of answering a challenge, and they get narrow keys to their own doors, which the doorman never learns about.
|
Slice authentication by who arrives, not by what serves the request. Humans are one audience - interactive, browser-borne, phishable, and they get a doorman: one strong flow, every human door in the estate behind the same desk, so door N+1 inherits the whole policy for the cost of a nameplate. Machines are the other audience - headless, scoped, incapable of answering a challenge, and they get narrow keys to their own doors, which the doorman never learns about.
|
||||||
|
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ So the controls live there, not on the origin:
|
|||||||
The origin's job shrinks to one thing: hold the tunnel open and serve. Everything hostile is filtered a
|
The origin's job shrinks to one thing: hold the tunnel open and serve. Everything hostile is filtered a
|
||||||
continent away.
|
continent away.
|
||||||
|
|
||||||
## The principle
|
## The safest port is the one never listening
|
||||||
|
|
||||||
Exposure isn't binary, and it isn't a synonym for port-forwarding. Push the perimeter out to an edge you
|
Exposure isn't binary, and it isn't a synonym for port-forwarding. Push the perimeter out to an edge you
|
||||||
don't host, make the origin speak only outbound, and the attack surface at home collapses to zero open
|
don't host, make the origin speak only outbound, and the attack surface at home collapses to zero open
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ the passphrase lives offline, not in any repository it unlocks, and the recovery
|
|||||||
machine and the key", assuming nothing else survived. A backup encrypted with a key stored next to it is a
|
machine and the key", assuming nothing else survived. A backup encrypted with a key stored next to it is a
|
||||||
padlock with the key taped to the shackle.
|
padlock with the key taped to the shackle.
|
||||||
|
|
||||||
## The principle
|
## Trust receipts, not reports
|
||||||
|
|
||||||
Trust receipts, not reports. Any system whose entire purpose is a future emergency - backups, break-glass
|
Trust receipts, not reports. Any system whose entire purpose is a future emergency - backups, break-glass
|
||||||
accounts, failovers, restore paths - is untested by definition on every normal day, so you have to
|
accounts, failovers, restore paths - is untested by definition on every normal day, so you have to
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ A checklist sounds bureaucratic until you notice what it changes: each item is a
|
|||||||
verify, and a failing item blocks the ship. "Pretty secure" isn't a state you can test for.
|
verify, and a failing item blocks the ship. "Pretty secure" isn't a state you can test for.
|
||||||
`grep` finding zero secrets in the bundle is.
|
`grep` finding zero secrets in the bundle is.
|
||||||
|
|
||||||
## The principle
|
## The site is the argument
|
||||||
|
|
||||||
The site *is* the argument. A platform engineer's portfolio should demonstrate the discipline it
|
The site *is* the argument. A platform engineer's portfolio should demonstrate the discipline it
|
||||||
advertises, and "it's a static page" is no excuse to skip the rigour - it's the cheapest possible
|
advertises, and "it's a static page" is no excuse to skip the rigour - it's the cheapest possible
|
||||||
|
|||||||
@@ -81,7 +81,7 @@ so. The wrong response is to call that a false alarm. It isn't. It's the system
|
|||||||
deliberate act, and the deliberate act is written down where the alert can be checked against it. The
|
deliberate act, and the deliberate act is written down where the alert can be checked against it. The
|
||||||
porter doesn't disable the alarm panel for the dark wings - he checks the panel against the log.
|
porter doesn't disable the alarm panel for the dark wings - he checks the panel against the log.
|
||||||
|
|
||||||
## The principle
|
## Design for off
|
||||||
|
|
||||||
Design for off. A node you can power down without ceremony is a node whose whole truth lives outside it - in documents, in git, in the log, and that is precisely the property you need on the day a node goes down
|
Design for off. A node you can power down without ceremony is a node whose whole truth lives outside it - in documents, in git, in the log, and that is precisely the property you need on the day a node goes down
|
||||||
without asking. The fleet you can put to sleep on purpose and the fleet that shrugs off losing a node in
|
without asking. The fleet you can put to sleep on purpose and the fleet that shrugs off losing a node in
|
||||||
|
|||||||
@@ -85,7 +85,7 @@ flows, and note the identity the traffic actually carries when it reaches the en
|
|||||||
`host`, `cluster`, a pod label, a CIDR. Then write the rule for *that*. A policy written from the
|
`host`, `cluster`, a pod label, a CIDR. Then write the rule for *that*. A policy written from the
|
||||||
architecture diagram is a guess; a policy written from observed flows is a fact.
|
architecture diagram is a guess; a policy written from observed flows is a fact.
|
||||||
|
|
||||||
## The principle
|
## Secure what arrives, not what was sent
|
||||||
|
|
||||||
A network policy is a doorman checking return addresses, and the postal system rewrites return
|
A network policy is a doorman checking return addresses, and the postal system rewrites return
|
||||||
addresses as a matter of course. You don't secure what was sent - nobody ever sees what was sent.
|
addresses as a matter of course. You don't secure what was sent - nobody ever sees what was sent.
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ Where it stands, honestly: three of the six nodes are staged and gated as of thi
|
|||||||
|
|
||||||
The flip itself is still blocked. Deliberately. One node is parked awaiting a hardware repair, so the door stays propped for everyone until that node passes its gate too. That isn't the plan slipping - it *is* the plan. The asymmetry does all the work: a staged lock you haven't turned costs nothing, while a turned lock with one unproven key costs a boot loop on the worst morning of the quarter. With arithmetic that lopsided, waiting isn't caution. It's reading the numbers.
|
The flip itself is still blocked. Deliberately. One node is parked awaiting a hardware repair, so the door stays propped for everyone until that node passes its gate too. That isn't the plan slipping - it *is* the plan. The asymmetry does all the work: a staged lock you haven't turned costs nothing, while a turned lock with one unproven key costs a boot loop on the worst morning of the quarter. With arithmetic that lopsided, waiting isn't caution. It's reading the numbers.
|
||||||
|
|
||||||
## The principle
|
## Failures that only surface at the next boot
|
||||||
|
|
||||||
Registry auth is one instance of a wider class: changes whose failure surfaces only at the next boot, restart, or failover - where the machine that would report the failure is the machine the failure takes down. For that class, make-it-and-watch is not a strategy, because there is nothing to watch until rollback is no longer on the table. The split is the strategy. Stage the change inert, in config, where it can be reviewed and reverted for free. Prove it per node, under conditions where failure is cheap and attended. Enforce only on full coverage - never on most of it.
|
Registry auth is one instance of a wider class: changes whose failure surfaces only at the next boot, restart, or failover - where the machine that would report the failure is the machine the failure takes down. For that class, make-it-and-watch is not a strategy, because there is nothing to watch until rollback is no longer on the table. The split is the strategy. Stage the change inert, in config, where it can be reviewed and reverted for free. Prove it per node, under conditions where failure is cheap and attended. Enforce only on full coverage - never on most of it.
|
||||||
|
|
||||||
|
|||||||
@@ -54,7 +54,7 @@ rather than inheriting a cluster default you can't see: `baseline` where root in
|
|||||||
(a VPN sidecar that needs `NET_ADMIN`, a time daemon that needs `SYS_TIME`). Run the rest at `warn` for the
|
(a VPN sidecar that needs `NET_ADMIN`, a time daemon that needs `SYS_TIME`). Run the rest at `warn` for the
|
||||||
restricted profile, and every apply tells you exactly which workload is one refactor away from the top tier.
|
restricted profile, and every apply tells you exactly which workload is one refactor away from the top tier.
|
||||||
|
|
||||||
## The principle
|
## Least privilege is a direction, not a pile of Roles
|
||||||
|
|
||||||
Least privilege isn't a pile of Roles. It's a direction: start by removing the access that nothing is using - the auto-mounted token first - then narrow what genuinely remains, then make the boundary enforce itself. Give
|
Least privilege isn't a pile of Roles. It's a direction: start by removing the access that nothing is using - the auto-mounted token first - then narrow what genuinely remains, then make the boundary enforce itself. Give
|
||||||
every workload its own identity, hand it no credential it doesn't need, strip it to the smallest process it can
|
every workload its own identity, hand it no credential it doesn't need, strip it to the smallest process it can
|
||||||
|
|||||||
Reference in New Issue
Block a user