AI CLI Tools, Provider Diversity, and the Consensus No Vendor Will Build You
I run several different AI CLI tools against the same work on purpose, not out of brand loyalty to any of them, and I've been doing it for real for over a year: multiple specialized agents, an independent Critic that rules pass or fail, production code shipping through it. This post is the concept and the setup, not a lab report, there's a short recipe near the end for checking the review step yourself.
Every AI vendor wants the same thing from you: stay on their model. That's not a conspiracy, it's just the business. Anthropic wants you on Claude, OpenAI wants you on GPT, xAI wants you on Grok. None of them has a roadmap line that reads "route 20% of calls to a competitor to catch our own mistakes." There's no product incentive for that. There's only a user incentive for it, and only if that user doesn't care which model is right, just that one of them is.
I do care about that, so I built it: different providers create, review, and rule on each other's work, held to a bar none of them set for themselves. Not a score, a ruling, pass or fail, with a reason attached that another person or another model can re-check. I've written about pieces of this system for over a year, but always from the outside, the hardware, the memory problem, the economics. This post is from the inside: the concept itself, how it actually works, and how I use it.
The industry already fights over who gets to train on whom, which is exactly why none of them would ever build this
I didn't have to speculate about whether vendors mind a competitor learning from their model: it's been a live fight. In August 2025, Anthropic revoked OpenAI's API access to Claude after discovering OpenAI was using it internally to benchmark GPT-5's responses against Claude's. Anthropic's terms ban exactly that: no access "to build a competing product or service, including to train competing AI models." OpenAI's defense was that benchmarking a rival is "industry standard." Both things can be true, and Anthropic cut the access anyway, its second such cutoff that year, after Windsurf during OpenAI's acquisition talks.
A messier, disputed version of the same fight is still running in 2026: a White House tech policy official publicly accused Moonshot AI of distilling Anthropic's Claude-Fable model to build Kimi K3, citing "millions of exchanges" identified by IP and metadata. I'm flagging that one as contested, not settled. Neither Anthropic nor the White House has published a query log, training manifest, or forensic analysis, and independent researchers argue the timeline doesn't work: Fable 5 had only been public a few weeks, not enough time to distill and train a model Kimi K3's size.
Worth being precise about what these fights actually are, so this post doesn't get lumped in with them: they're about training a rival frontier model on another lab's outputs at scale, without permission, straightforward IP extraction. Every major lab's terms ban it; OpenAI even sells a distillation product, restricted to teacher=OpenAI, student=OpenAI. What I'm doing in this post is categorically different: I'm a paying customer of several separate services, asking each one the same question and comparing their answers. That's not distillation, it's comparison shopping.
But notice what that proves beyond "no incentive." These aren't neutral competitors who simply haven't gotten around to building a shared comparison tool. They're actively revoking each other's API keys and getting the White House involved, in the same eighteen months. A consortium of Anthropic, OpenAI, and Moonshot jointly shipping a product that routes your question across all three and tells you when they disagree isn't a roadmap gap waiting to be filled. It's organizationally impossible between parties that ban each other's access and accuse each other of IP theft at a federal level. There is no boardroom where that feature gets greenlit. The only place a tool like that can exist is outside all of their walls, built by someone who's a paying customer of all of them and a competitor to none, which is exactly the position an end user, and only an end user, is in.
I'm not claiming this is new
I said as much the first time I put names on these roles, back in December 2025, running a four-role review loop (ProjectManager, Developer, Reviewer, and an adversarial critic I called "Lucifer" at the time, because it was the devil's advocate point of view, now I just call it the Critic) on local Ollama models:
"This is not novel in human teams. It's basic team dynamics. What's novel is making it work with open source models that fit in a local machine."
Dual-approval review teams and adversarial gates aren't my idea, and there's a real research lineage behind the pattern: MetaGPT (Aug 2023) assigned PM/Architect/Engineer/QA roles to GPT instances over a year before I named mine; AutoGen (Microsoft, same month) generalized multi-agent conversation as infrastructure; and a steady line of Multi-Agent Debate research since has been studying whether structured disagreement between models actually improves output over a single model's answer.
The closest match I found is current: an ICML 2026 DL4C workshop paper, "Adversarial Review: Structured Disagreement for Grounded Agentic Code Review," built around almost exactly my shape, a coding agent, a Reviewer, and a Critic who audits the Reviewer's findings through structured disagreement rather than trusting them. Their own stated motivation is the one my Critic's seat file was already built around, in their own words: "when two LLM agents are asked to agree on a joint output, they tend to agree with each other. They do not always find the truth." I use three roles throughout this post because three is the simplest version that demonstrates the mechanism, not because any specific benchmark says three is optimal; more on how far you can scale that past three, under "How it's actually built, briefly."
Why this works
Four things have to be true for a multi-model review loop to beat a single model, and I can point to a reason for each one instead of just asserting it. One more thing closes out the section afterward, not a fifth requirement, just the historical note that none of this is actually new.
Different labs make different mistakes. A model reviewing its own output, or output from a model trained the same way on similar data, shares its own blind spots with what it's checking. That's the whole reason to use multiple providers, DeepSeek, Zhipu's GLM, and Moonshot's Kimi for example, or Claude, or Grok, instead of three calls to the same model: a Reviewer with a genuinely different training lineage is more likely to notice something a same-lineage reviewer would also miss.
The persona has to be disciplined, not just adversarial. "Be adversarial, find bugs" sounds like the right instruction and isn't. A model told to find problems will find problems, real ones and manufactured ones, with no way to tell you which is which. A model told to prove every claim against a stated scope, and told explicitly not to penalize what's outside that scope, produces a ruling you can act on instead of a pile of busywork.
Re-verification has to be independent, not inherited. A Critic that reads "Reviewer says it's fine" and agrees has added nothing; it's just a second vote for the first model's blind spot. A Critic that re-derives the claim itself, from the code, and only then compares notes with the Reviewer, catches the specific failure where a Reviewer's reasoning was wrong but its conclusion happened to be right anyway. This isn't just my own rule, it's written into the team law every seat in my setup runs on: "if Developer, Reviewer, and Critic all agree without friction, look harder. Structured disagreement > comfortable consensus." The Critic's own file says the same thing in its own words: "When all three of us agree, look harder, that is when groupthink is most likely." Agreement is the moment to get more suspicious, not less, because three independent reads landing on the same wrong answer is a real failure mode, not a reassurance. This isn't just theory: the workshop paper named above, studying this exact reviewer-critic pattern, found naive agreement scoring worse than a single reviewer alone (F1 0.457 vs. 0.495), until the critic was forced to ground every disagreement in cited code, which pushed it to 0.533, the best of the methods tested. An undisciplined second opinion can make review worse than no second opinion at all.
The gate has to be a ruling, not a feeling. "Looks good to me" doesn't compose across models and doesn't survive someone asking "compared to what, measured how, what would prove this wrong?" A pass or fail that has to name its reasoning, and gets rejected and sent back if it can't, is falsifiable in a way "looks fine" never is. It's also the only version of "the team approved this" that another person, or another model, can actually audit after the fact, because there's a stated reason to check, not just a vibe.
As a note, this isn't new to engineering, it's just new to applying it to AI. Humans already run on this: when two people give different versions of the same story, the truth is usually somewhere in the middle, and that's exactly why no developer certifies their own code for production. The formal name for it is separation of duties, requirement, design, review, and production shouldn't sit with one person, and it's slower on purpose. The counterargument is real too, and worth taking seriously rather than ignoring: one study found overly specialized software teams can produce worse output once people stop sharing knowledge across tasks, which is part of why the industry also moved toward embedded QA and "you build it, you run it." The lesson isn't "more separation is always better," it's that the separation has to be structured, not just present, which is what an independent, falsifiable ruling is doing for a team of models instead of a team of people.
How it's actually built, briefly
This is architecture, simply put:
Picture a TechOps team: different people, different roles, a shared ticket queue. Swap "different people" for "different models" and the ticket queue for a ledger, and that's the whole architecture. The role, Developer, Reviewer, Critic, whoever, is a folder and a system prompt; the model sitting in it is the one thing allowed to change without anyone else noticing. That's not a metaphor, it's literally the folder layout, ten seats, each with a real AGENTS.md and a CLAUDE.md that's a 9-byte symlink to it (one file, two names, so Claude Code's convention and every other harness's convention both resolve to the same instructions):
AI_TEAM/: AGENTS.md, CLAUDE.md -> AGENTS.md
CRITIC/, DBA/, REVIEWER/, and 7 more seats, each the same two filesThree layers do three different jobs. Mind is the durable part: an append-only coordination ledger (a database trigger blocks UPDATE and DELETE at the engine level, trg_events_no_update/trg_events_no_delete, verified live) plus a separate knowledge store with three views, MySQL for structured facts, Neo4j for relationships, MemPalace for semantic recall. Cognition is a single model call, reasoning over whatever's in Mind, then ending; it has no memory of its own. Display is what a human watches through, a separate CLI channel per seat for me day to day, without becoming the coordination mechanism itself.
Seats are independent Cognition sessions, woken on their own schedule, reading and writing the ledger through MCP instead of a hand-off from one orchestrating thread, which is both what makes it decentralized and what makes it scale: adding a fourth or fifth seat means pointing another folder at the same ledger, not building new plumbing.
How I actually use this
This isn't a system I stood up to write a post about. It's been running for over a year, and the seats aren't a demo roster, they're doing real work: several Developer agents, not one, each with a narrow lane, a DBA agent for schema and migrations, a Security agent for auth and secrets, each with a sharper, narrower prompt than "review everything." JavaScript and Python work go to different Developer seats on purpose, because a model that's spent its whole context on one language's idioms writes better code in that language than one switching between three.
The CLI control isn't a feature I'm describing in the abstract, it's how I spend my day: a separate terminal per seat, and from each one I can run a seat on auto, step it one iteration at a time, or interrupt it mid-task, depending on how much I trust what it's doing right now. The gate applies to whatever's actually shipping that day, not a toy example picked for a blog post: the Critic rules pass or fail, with a reason, before anything goes in.
What a year of this has actually bought me: fewer surprises after something ships, and code I trust more by the time it clears the gate. That's not a number I can back with a single command, it's a year of living with it, which is a different kind of evidence than anything else in this post and I want to be clear about that difference rather than blur it. If you want to see the mechanism itself rather than take my word for the year, here's how to do that with whatever you've got.
Run this yourself, with whatever you've got
None of this needs my config, my ledger, or my seat files. It's three prompts and a loop. You don't need Ollama or any specific models; you need three genuinely different providers, any three, and a way to call each one with a system prompt. Here's the actual shape, trimmed down.
Pick a task with a clear, checkable scope, something you can run and look at the output of. Then three system prompts:
DEVELOPER:
"No placeholders, no 'fix later.' Quality over speed.
Write the code. Don't review or gate your own work."
REVIEWER:
"Never assume the Developer is correct. Prove every claim.
If you can't re-run it and see the output, it's not approved.
Don't penalize anything outside the stated scope.
End with exactly one line: VERDICT: PASS or VERDICT: FAIL, plus the reason."
CRITIC:
"Never guess, always validate. Every claim is a hypothesis
until you've independently re-derived it, not just trusted
the Reviewer's sign-off. Ask: compared to what? measured how?
what would failure look like? Don't rewrite the code, only rule.
End with exactly one line: VERDICT: PASS or VERDICT: FAIL, plus the reason."Then the loop, plain language, not a specific framework:
Developer writes a draft.
Send draft to Reviewer.
If Reviewer FAILs it: send the specific reason back to Developer, repeat.
If Reviewer PASSes it: send draft + Reviewer's reasoning to Critic.
If Critic FAILs it: send Critic's reason back to Developer, repeat.
If Critic PASSes it: done.There's no percentage to tune at the gate itself. The Critic's ruling stays pass, fail, and why, not a score crossing a line. Numbers do show up a layer down once you automate the deliberation that feeds the gate, agreement ratios, dissent counts, that kind of thing, in other automated pipelines I run, and that's fine; a number deciding whether a sub-step converged is a different thing than a number deciding whether the Critic approves. Don't let the second one quietly replace the first. If you automate this, cap the loop at a set number of rounds before raising it back to you, so a bad run doesn't repeat forever and run up a bill. Use at least three different models from three different labs, not three calls to the same one; the whole mechanism depends on that. Beyond that, the specific providers don't matter; I used Ollama's cloud tier because it was the cheapest way to reach DeepSeek, GLM, and Kimi from one script, but the same three prompts work against three API keys, three browser tabs, or three different vendors' free tiers.
Where this sits
This is my opinion on how AI should get used, for development, but for anything else too: if you lean on just one provider, for code or for a simple chat question, you're trusting only that provider's training data and whatever blind spots come with it. I don't, and my own workspace reflects that, a concept, a reason for each piece of why it works instead of just an assertion, and a short recipe for checking the review step yourself instead of taking my word for it. That's less dramatic than "AI team builds better code than any single model," and it's also the only version of the claim I can actually back, a year of living with it, kept separate from the parts that are still just argument rather than blurred together. None of it came from a vendor, and it couldn't have: this exists because someone outside all of their walls had a reason to build it that none of them do. Take the three prompts under "Run this yourself," point them at whatever three models you've already got, and see what it catches on something of yours. That's the only test that actually matters to you.
Related reading
- The Homelab That Just Kept Growing (Part 1): the hardware and the original memory problem this whole series starts from
- The Homelab That Just Kept Growing (Part 2): the Mind / Cognition / Display architecture this post is built on
- Open Source AI Models: Building a Development Team: where Lucifer, the dual-gate, and the line I still stand by first showed up
- I'm Trying to Build a Mind on Hardware I Own. I'm Not There Yet.: the honest scorecard on the digital-mind project this architecture also feeds
- I Kept Paying for One AI Model. Then Ollama Handed Me 37 for Twenty Bucks.: the economics that make running three providers side by side this cheap
- Running Claude Code Against Anything: profiles, isolation, and why tool and model are independent dials
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review (ICML 2026 DL4C workshop): the closest academic match to this setup's Developer/Reviewer/Critic shape, independently arrived at
- MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework: the 2023 paper that got there first with named software-engineering roles
- Anthropic revokes API access to OpenAI's Claude service for violating terms of service: the receipts for "no provider wants this built"
