Back to blog

Making three AIs referee each other

Before I ship a redesign on Blink, I don't just ask Claude if it's good. Claude wrote the thing, or reviewed the plan for it, so asking it to grade its own homework is close to useless. It's invested in its own reasoning by construction. What I do instead is put the work in front of ChatGPT, Gemini, and Grok, and make them argue about it.

Mechanically it's dumber than it sounds. I have a Claude session drive an actual browser, not an API, and paste the same design canvases into all three chat interfaces. Same five screens, same prompt, three separate conversations. Then it pulls the responses back out and I read all three side by side.

Why not just ask one model harder

Because a single model's review is a single point of view wearing the costume of an objective verdict. Ask GPT whether your onboarding flow is good and you get an answer shaped by whatever it tends to notice, and no way to tell if that's a real gap or just its house style. Ask three, and the review stops being "what does this model think" and starts being data.

The clearest case: I'd designed the navigation for Blink's redesign as gesture-only, swipe between the main views, no visible tab bar. Clean, I thought. Confident. All three models flagged it independently, unprompted, as a trap for exactly the audience Blink serves: people already fighting focus and working memory, now expected to remember a gesture vocabulary nobody told them existed. When three models that don't talk to each other converge on the same critique without being led there, I stop treating it as an opinion and start treating it as a finding.

That's the useful signal: unanimous convergence. If Claude, ChatGPT, Gemini, and Grok all land on the same problem from different angles, the odds that it's a shared blind spot in all four are much lower than the odds that one model just made something up.

The catch that only one model made

The unanimous ones are the ones I quote in standups. The valuable ones are rarer and weirder. I ran the same panel treatment on a delivery plan, not a design, an execution plan, asking each model to check whether it actually verified what it claimed to verify. Two of the three walked through it and said, essentially, looks solid, the plan covers pushes, UI states, and user journeys.

One model didn't say that. It said the plan verified pushes, UI, and journeys, but never verified that the analytics events firing underneath all of that were actually correct. Not "did the event fire," which the plan did check, but does it carry the right properties, does it fire once instead of three times, does the funnel math it feeds actually mean what the dashboard says it means. Its own phrasing stuck with me: you do not verify measurement. That's the whole reason to run three models instead of one. Had I only asked the two that missed it, I'd have shipped a plan with a hole in exactly the place you only notice once the numbers are already wrong and nobody can say why.

They don't just disagree, they think differently

Past a few rounds of this you start to notice each model has a lens, not just an opinion. One consistently reasons in retention psychology, what makes someone come back tomorrow. One reasons in state machines, what happens when this screen is reached from an unexpected prior state, what's the invalid transition. One ranks everything by severity before saying anything else, like it's triaging a bug tracker. None of these is the correct way to review a product. Together they cover more ground than any one lens would, including the one I'd have picked myself.

The unglamorous part is the browser, not the models

The part of this that actually eats time isn't the AI reasoning, it's getting text in and out of three chat UIs reliably. One session had a model's page render its response in a virtualized DOM, the kind that only paints the rows currently scrolled into view, and my extraction read back ninety characters from what was actually a five-thousand-character review. I nearly concluded the model had nothing useful to say and queued up a re-ask. The review was sitting right there the whole time. I'd just scraped a sliver of it. I only caught it because the reply felt too short for the prompt I'd sent. That instinct, not any error message, is what saved the re-ask.

That's the honest tax on this technique. Driving three separate web UIs through a browser is fussier than any API would be, and the failure modes are dumb ones: a stale selector, a lazy-rendered pane, a model that asks a clarifying question instead of answering. None of that is exotic AI failure. It's just automating a browser, with all the brittleness browser automation always has.

Same panel you'd assemble for humans

None of this is a new idea dressed up in AI language. If I were shipping something high-stakes with a human team, I wouldn't want one reviewer either. I'd want a designer, an engineer, and someone from support who's heard every user complaint firsthand, because each would catch something the others wouldn't. Models are the same. Get a few different ones, look for where they agree, and don't discard the one that says something the others didn't. That's usually the finding worth the whole exercise.