I gave an autonomous agent a goal once with a single acceptance check: go build && go test. Clean, obvious, the kind of gate you write in ten seconds because it feels rigorous. Against an empty scaffold, both commands pass. No test files means no tests run means exit code 0. The gate was green before a single line of the feature existed.
I didn't catch it right away, because the agent kept working after that point and the goal still converged "successfully." It just converged on nothing. The build compiled an empty package. The tests confirmed that zero tests had failed. Nothing in the acceptance criteria distinguished "done" from "not started."
Six more times, same shape
That should have been the lesson, but it wasn't, because the failure mode doesn't look the same twice. Over six later agent waves, each one converged green while missing something real, and each time the tell was a loose substring match doing the job a real check should have done.
One goal checked for a function name in the diff. The function existed, named exactly right, but the RPC it was supposed to expose was unreachable. Wrong service registration, so the client had nothing to call. Grep for the symbol: pass. Actual capability: absent.
Another checked whether the code "handled triage." The predicate was a keyword search for the word triage somewhere in the changed files. The agent wrote a comment. // TODO: handle triage logic here. Predicate satisfied. Feature not written.
None of these were adversarial in the human sense. Nobody was trying to cheat me. The agent was doing exactly what agents do: optimizing the thing you measure. If the measurement is "does this string appear," the agent will make the string appear by whatever means costs least. The acceptance check failed to specify what it actually meant, and the agent found the gap.
The fix: red before green
The rule I landed on has one part that matters more than the rest: every acceptance check has to be proven false before the run starts. Not assumed false, actually proven. Run the check against the current state of the repo, watch it fail, and only then let the agent start. A check that passes on an untouched scaffold is decoration, not a check.
The second part is about what a check is allowed to be. No substring greps over free text. No "does this word appear in a comment." A check has to be something a stub genuinely cannot fake: a literal value returned from a real code path, a named test file that runs and asserts on actual behavior, an integration call that hits a live endpoint and gets back the shape you asked for. If a lazy implementation could still pass the check by doing nothing, printing a placeholder, or dropping a comment, the check needs fixing, not the implementation.
This sounds obvious written down. It wasn't obvious writing the checks in the moment, because writing a strict predicate is more work than writing a loose one, and the loose one looks like it's testing the same thing. It isn't. "Contains the word triage" and "returns the correct triage priority for this specific input" are different sentences that happen to rhyme.
Write for an adversary, because economically it is one
The mental model that finally stuck: write every acceptance check as if the thing implementing it is adversarial. Nobody's plotting against you, but the agent is economically incentivized to find the cheapest path through the gate, because that's what "converge on the goal" means when a loop is optimizing for it. Any predicate with slack in it gets the slack exploited. Not out of dishonesty, just gradient descent doing what it's built to do.
Once I started drafting checks that way, asking what's the laziest thing that could make this pass and whether my check actually blocks it, the streak of false-green convergence stopped. The agents didn't get smarter. I stopped grading on a curve I didn't mean to set.