Every deploy on Blink runs under an IAM role. That role has a permissions boundary attached to it: a ceiling bolted above its actual permissions, one it can never punch through no matter what policy gets attached later. The point is simple. If CI ever gets compromised, the attacker inherits whatever the role can do, and the boundary caps that at something survivable. It's a good pattern. I didn't invent it, I just leaned on it.
A security review turned up a gap in that ceiling. The role could still modify its own boundary, which defeats the entire purpose. A boundary you can rewrite from inside isn't a ceiling, it's a suggestion. So we scoped a fix, stripped the role's permission to touch that one policy resource, and shipped it like any other change.
The fix that couldn't deploy itself
Next deploy failed in eight seconds. Not a flaky-test eight seconds, a something-is-structurally-wrong eight seconds. I pulled the logs and the shape of it was almost funny. Applying the fix means updating the boundary policy. The fix, once live, forbids the role from updating the boundary policy. So the deploy that ships the fix needs the exact permission the fix revokes, and by the time infra tries to apply it, the role no longer has it.
The fix couldn't deploy the fix. Not "the fix has a bug." The fix was correct, and correctness was the problem. It worked so well it locked out the only actor that could have shipped it.
I want to say I saw it coming. I didn't. It reads obvious in hindsight: of course a role can't grant itself permission to remove its own permissions. But when you're scoping a boundary fix you're thinking about the attacker, not about your own CI pipeline being the next thing to get denied.
Applying it by hand, and the plot twist
First move: apply the policy change once, by hand, with admin credentials. That's the standard escape hatch for this kind of deadlock. A human with a higher trust tier does the one action the automated path can no longer perform. It worked. Boundary updated, role locked down, deploy green.
Except the next deploy failed the exact same way. Same eight seconds, same denied action, same resource. I checked whether the boundary had somehow reverted. It hadn't. What was actually happening was worse in a boring way: our infra tool re-diffs that policy on every plan, and something in how it normalizes the JSON meant the computed diff never fully settled to zero. Whitespace, key ordering, some canonicalization quirk. The two representations were semantically identical and syntactically not, so the tool kept proposing a no-op change to the one resource the role was now forbidden to touch. On every run, indefinitely.
Applying it by hand hadn't fixed the pipeline. It had fixed the policy and left the pipeline armed to fail the same way on a schedule.
The durable fix is a declaration, not a patch
The real fix wasn't another permission tweak. It was taking that resource out of CI's plan entirely, an explicit exclusion so the tool never computes a diff against it, flapping or not. The boundary policy is now owned out-of-band: changed only by a deliberate human action outside the deploy pipeline, never something CI reasons about at all.
That's a different shape of solution than "grant the right permission." It's "stop asking the automated system to have an opinion about this resource." The ceiling needs an owner who isn't subject to the ceiling. Otherwise you've built a system that can lower its own guardrails but never raise them back, or one that gets stuck re-litigating a settled decision on every run because a JSON serializer doesn't agree with itself.
The lesson isn't "test your security fixes more." We did test it. It worked, right up until it worked so well it walled off its own repair path. The lesson is that any system enforcing least privilege needs a designated out-of-band owner for the constraint itself, one that sits outside the blast radius of the constraint. "I applied it once by hand" feels like closure. It isn't the whole fix. If the automated path can re-trigger the same lockout on its next run, you haven't fixed the process, you've delayed the next incident by one deploy cycle.
Least-privilege is supposed to make the failure mode smaller. It does, right up until the privilege you took away is the one your own pipeline needed to keep functioning. Then the smaller failure mode is a pipeline that fails identically, quietly, on every run, and nobody notices until they go looking for why deploys keep dying in eight seconds.