Ask who checks whom

Before a company lets AI agents into its code or its customer service, the board usually asks which model it runs on. I defend the thesis that, among frontier models, the answer settles less than it seems. Whether the deployment will be safe and whether it will pay off is largely decided by the control layout: who plans, who challenges, who approves and what number has to be beaten before the work even starts.

I am arguing with the assumption behind the Claude-versus-GPT comparisons: that the choice of model is the main decision of a deployment and that a benchmark table is the right way to make it. I do not attribute this assumption to any company. My quarrel is with the reasoning. The model matters. I will show when it decides the outcome. My claim is only that a buyer of an AI service should ask about it later and about the control layout earlier.

A sprint whose model is the least copyable part

On 23 September 2026 Raymond Wang and two co-authors described on the claude.dev blog a two-week August sprint by an Anthropic team. The goal was simple: to speed up the company's own apps. The post is titled How we made claude.ai 3x faster in two weeks. The headline number concerns a single measure. At the 75th percentile, the time to a page you can already type in, on a fresh load of claude.ai, fell from 3.1 to 0.55 seconds. That is a 5.6-fold improvement.

This number is easy to confuse with another. Before and after the sprint the authors measured thirteen values across four user journeys. The geometric mean of the improvement came out at 3.1 times, hence "3x faster" in the title. The match between 3.1 seconds and 3.1 times looks coincidental. If you see the sprint summarised as "claude.ai loads three times faster", the summary has mixed up two measures.

More interesting than the result is how it was reached. Claude worked in a Slack channel through Claude Tag, a product in beta, on an internal research model the authors describe as roughly comparable to Opus 5.5. Every thread of work had a named owner, a person. When a change was visible to users, Claude showed before-and-after screenshots or recordings, and the thread's owner ruled on the result.

Every pull request went through automated review and at least one human approval. Unit tests came before optimization, not after it. Anything that could cause a visible problem shipped behind a short-lived feature flag. High-risk changes went first to employees, then to one percent of users, and only at the end to everyone.

One element has an English name: a ratchet. The team added to continuous integration (CI) two measurements of instruction counts on selected paths. From then on every pull request that raised that count failed CI, and a daily job lowered the ceiling every time the count dropped even a little. The number could only go down.

According to the authors, the team merged more than three thousand changes this way without a single customer-facing incident or rollback, but this is Anthropic's own claim, without independent confirmation. I take it as a description of the process, not as proof that it works in every company.

Let us now ask as a buyer. What from this sprint carries over to your organisation? You cannot carry over the model, because it was internal and cannot be bought. The post also does not let us separate the credit of the model from the credit of the layout. That has to be said honestly. What it does let us see is which part of the process can be copied, though with no promise of the same result: the thread owner, the number with a ratchet, human approval and staged rollout.

A convenient axis of dispute

A buyer compares models because the comparison gives them a number that is easy to defend before the board. This is my interpretation, not a research finding, but it is easy to test on yourself. A benchmark table is someone else's measurement on someone else's task. It has one advantage: it is ready. A control layout has to be designed yourself, and you have to take responsibility for it.

Professions that have long lived off judging other people's work learned to manage this problem without language models. The code of ethics of the International Ethics Standards Board for Accountants (IESBA) lists five categories of threats to compliance with ethical principles, objectivity among them. One of them is the self-review threat: the risk that a professional will not properly evaluate the result of earlier work, their own or that of someone in their firm. An auditor is not chosen by IQ. It also matters whether they are judging their own work.

Now the strongest objection to my thesis. The model really does matter. Anthropic's data show it. On 9 April 2026 the company described on the claude.com blog the advisor strategy: a cheaper model carries out a task from start to finish, and a more expensive Opus advises it at decision points. On the BrowseComp test, Haiku with an Opus advisor scored 41.2 percent against 19.7 percent for Haiku alone. This pair costs 85 percent less per task than Sonnet, but trails Sonnet alone by 29 percent in score (the source does not say whether this means percentage points). These are benchmark results reported by Anthropic, not a production measurement.

These numbers say something uncomfortable for both sides of the dispute. A model's capabilities set a ceiling that review and approval alone will not raise. If a task exceeds what the executor can do, review will show the failure sooner, but will not prevent it. The opponent is also right when the volume is huge and the cost of a single task decides profitability, because the choice of model then becomes a first-order financial decision.

The same numbers show a second thing, though. The same Haiku more than doubled its score when it got a second participant at decision points. The executor did not change. What changed was who speaks before a decision, and that was a stronger model. This example supports both sides of the dispute at once. In the Sonnet variant the advisor added 2.7 percentage points on the SWE-bench Multilingual test and cut the cost of a task by 11.9 percent. The advisor, as the authors write, never calls tools and never produces user-facing output. It only advises.

So I narrow the thesis. One caveat: the advisor data concern models of different strength, so they do not test the narrowed thesis directly. For a task within reach of several frontier models, the question about the model stops being the first question. The first becomes who checks that model and against what number. For a task at the edge of capability the order reverses and you start with the model.

Three roles that one executor should not combine

Let us start with the roles. The planner breaks a goal into steps and acceptance criteria. The challenger looks in the plan and in the result for errors the planner does not see. The approver has the right to say "no" and bears the consequences of their "yes". The fourth role, the executor, is sometimes separate and sometimes merged with the planner. The first three should not be held by one person, or by one model that sees the same as the planner.

The reason is not aesthetic. In 2023 Jie Huang's team published Large Language Models Cannot Self-Correct Reasoning Yet, later accepted at ICLR 2024. The authors showed that large language models struggle to correct their own answers without outside information. In some tasks the result after self-correction was worse than before it. The study concerned reasoning tasks. I do not carry it over directly to every agent's work, but it is enough not to assume that a model will catch its own errors.

The second paper goes further. In a 2024 paper, LLM Evaluators Recognize and Favor Their Own Generations, Arjun Panickssery and colleagues described an evaluating model's tendency to score its own texts higher. Models such as GPT-4 and Llama 2 told their own texts from others' with non-trivial accuracy, and in fine-tuning experiments the strength of this preference grew linearly with the ability to recognise themselves. Its direction matches what auditors have long known about the self-review threat.

I derive a practical rule from this, though it is my recommendation, not a research finding. If the same model plans, checks and approves, you have one point of view in three costumes. I assume that a fresh session of the same model removes the shared context of the conversation but not the tendencies the model has from training. So the challenger should, where possible, come from a different family of models, or at least get different input from the planner, for example the requirements alone without the planner's reasoning. The approver, at least for irreversible decisions, should be a person with a veto.

The tooling market has started to offer this split as a ready pattern. Chase AI released the claudex-loop repository under the MIT licence: a skill for Claude Code that can also run in Codex. It works in four phases. After scoping and writing down requirements with acceptance criteria, the plan goes to review by a second provider. The review loops until the reviewer approves the plan, blocks it or the round limit runs out, five by default. Whoever built the code does not grade it: the inspection is done by the second model in a fresh session.

In the repository I found one result from use. The first full planning run for a CRM system produced 55 findings in five rounds, and the author calls this an illustration, not a controlled measurement. So we do not know how much claudex-loop reduces errors. We know which layout he thought worth building.

A more interesting signal came from OpenAI. The company published in the openai/codex-plugin-cc repository an official plugin that lets you call Codex from inside Claude Code, a competitor's tool. The announcement on the OpenAI developer forum dates from 30 March 2026. The plugin has a command for an ordinary review, a command for an adversarial review that challenges design choices, and an optional gate. The gate runs a Codex review when Claude finishes its work. The authors openly warn that it can drain account limits quickly.

The reel that showed the plugin added on its own that models are by nature terrible at checking their own work. I did not find that sentence in any OpenAI source. The bare fact remains, and it is enough. When a lab builds a tool in which its model reviews the work of a competitor's model, the split of roles stops being the idea of one independent creator. A model provider offers it, and in a rival's tool at that.

The advisor strategy splits roles within one family of models, except that it splits authority, not the judging of the result: the executor acts and the advisor can only advise. To challenge a result I would still reach for a model from another family, though this is my hypothesis, which none of the cited authors measured. So a split of roles does not always require a split of providers. It requires a clear answer as to who may change the state of the system and who may only raise an objection.

Splitting roles costs money. Every review round is another model call and more tokens. claudex-loop has default round limits for this. Plan review has five rounds by default. After the build there are two fix attempts, and inspection gets two rounds. The OpenAI plugin warns about draining account limits quickly. A challenger without a round limit can circle in a loop longer than the task is worth.

The cost does not have to rise, though. In Anthropic's test the pair of Sonnet with an Opus advisor came out 11.9 percent cheaper per task than Sonnet alone, even though the advisor is the more expensive model. The source says the advisor writes a short plan, typically 400–700 tokens, billed at Opus rates. It does not break the saving down into components, though. The conclusion is mine, and it concerns the challenger, not the advisor: when you design the layout you also decide at what moment the challenger speaks. Review at every step and review before an irreversible decision are two different budgets.

A number set before the change

Three roles without a number give a tidy process that does not know where it is going. The last element of the layout is a measurable goal and a baseline measurement, set before the agent starts changing the system. In Anthropic's sprint the team began with instrumentation. It set its targets from Claude's own estimates for about twenty selected projects. It hit twelve of the thirteen targets by the third day and then kept setting new ones. In another company the goal will be the time to handle a ticket or the cost of one task in tokens.

Without a number an agent optimises for what can be judged by eye. This too is an interpretation, but easy to observe. When the only test is a reviewer's impression, the work converges on a text that looks good in review. A number measured in production does not share that impression.

The claude.ai ratchet turns a number into a gate, though it guarded a proxy measure: instruction counts on two code paths. One assembles the message tree, the other scans the status line in Claude Code. Nobody had to remember that number, because CI rejected every change that raised it. An instruction counter will not tell you, though, whether a new screen layout is readable for users. For visible changes a person ruled anyway, on the basis of before-and-after recordings. The number guards what can be measured, and the thread owner guards what the counter will not see.

The takeaway is simple and uncomfortable. Before you hand a process to agents, write down one number, how it is measured and its baseline value from recent weeks. If you cannot state that number, the deployment is not ready whatever the model. In six months probably nobody will be able to say whether the agents improved anything.

The approver who stops reading

On the busiest days of the sprint, the authors write, more than two hundred changes landed, and each got at least one human approval. The authors do not say how many people approved changes or how long one took them.

This number leads to an objection that has to be raised openly, if hypothetically. If one person approved two hundred changes a day, at some point they would start clicking without reading. The approval stays in the process. It loses its content. The approver's role then turns into a rubber stamp, and a rubber stamp will not catch an error that has already slipped past the planner and the challenger.

Anthropic's post does not claim that human approval was enough. Next to it worked the CI ratchets and the feature flags. High-risk changes went to employees first. I read this as a layout in which safety does not depend on one person's attention in the last hour of the working day. A rise in instruction counts on the two guarded paths will be stopped by the counter in CI. In a high-risk change, an error a person missed reaches employees first, then one percent of users, before it possibly reaches everyone. This is my interpretation. The authors do not put it this way.

My log lets me count the same thing on a smaller scale. Over 22 days agents used 208 of my consents, an average of just under ten a day. Each was created after my "yes" in the conversation, though at least one known consent was used by the wrong session. At ten a day this works. At two hundred I would not read.

The conclusion for whoever designs the layout is concrete. Put a human gate where the decision is irreversible and the number of such decisions lets you read each one. Everywhere else let the number, the flag and staged rollout do the guarding. An approver buried in requests stays on the org chart in name only.

My layout: a bridge between two agents and a guard

I will describe the layout I work in myself, because its failures say more about control topology than its successes, and I have each of them recorded in the log or in the rules written after it. Day to day I work in Claude Code. A second model, OpenAI's Codex, carries out reviews through the plugin mentioned above. Both tools sign in with my subscriptions. In this circuit I do not pay for API calls by the token.

The most important part of the layout is not a model. It is a guard script that runs before the commands an agent issues. It blocks irreversible operations, such as a forced push, deleting files and deploying to production, by default. Until 11 October it also guarded pushing to the main branch and merging pull requests through GitHub. A second guard then took over those two operations. When this script itself crashes, it lets the command through and logs the crash, so this gate too has a gap. It lets a blocked operation through only when a one-time consent token exists, which is created after my explicit "yes" in the conversation. The token is valid for 30 minutes. It is used up on first use.

The guard writes every decision to a log. That lets me give numbers instead of impressions. From 25 September to the evening of 16 October 2026 it recorded 776 blocks. Among them were 97 irreversible git and GitHub operations, 45 recursive or forced deletions, 9 attempts to deploy to production and 6 attempts to print a file with secrets. Until 11 October the rule also covered pushes to the main branch and merging pull requests through GitHub, so some of those blocks may have concerned them. Over the same period a consent token was used 208 times. The numbers cover all my projects and all agent sessions. They come, though, from one log only. The second guard keeps a separate log and is not part of these numbers.

The largest group of blocks surprised me. As many as 346 of the 776 concerned recursive searches with a time limit. The guard blocks them because an empty result from such a search does not prove that something is absent. The search may simply not have finished in time. This rule appeared after an agent announced that a project lacked a certain function, although it was there. About 45 percent of the blocks therefore protect the quality of evidence rather than the safety of data.

What went wrong? On the ninth of September a merge landed on the wrong target branch. Earlier, in August, a diagnosis was made on a stale copy of the repository and described a state that no longer existed. I turned each of these events into a rule: before merging, check the target branch; before diagnosing, fetch the latest state. I wrote the rules down where the agent reads them in every session, and the guard enforces the most important ones.

The most instructive slip happened on 11 October. I was working in two sessions in parallel on two projects. A consent token I had created for one session was used by the other. The consent went to someone else's operation. The fix was to bind the token to the session ID. A consent given to one executor stopped working for another.

This incident shows well how control differs from its appearance. I had the approver's role and I had a gate. What was missing was binding the consent to a specific executor. When there is more than one agent, every consent needs an addressee. In a company the counterpart of this rule is the question of whose permissions an agent inherits when it acts on behalf of an employee.

What remained? Human approval of irreversible operations, a log of decisions and the rule that the numbers in a report come from a log or a measurement. None of these depends on which model happens to be working in the terminal.

Scale: when a central gateway makes sense

On 1 October 2026 Alok Srivastava and co-authors described on Uber's engineering blog the MCP Gateway platform. MCP is the Model Context Protocol, through which agents call tools and services. Uber has thousands of internal services. Individual teams built integrations for agents independently of one another. The result was fragmented tooling and duplicated infrastructure. The authors write that without a shared architecture, scaling MCP would increase operational complexity, security risk and friction for developers.

The gateway now hosts more than 800 MCP servers and more than 5,000 tools. Authorization works at the level of a single tool. More important than the scale, though, is one rule. Every server and every tool starts disabled and has to be reviewed and enabled by the team that owns it. The authors frame this as a design principle: discovering a tool does not mean exposing it. Every change to a tool's description requires approval by the server's owners.

Uber is right for Uber. With thousands of services and hundreds of teams, a central gateway replaces many separate integrations with one infrastructure, and the decision to enable a tool still lies with its owner. A twenty-person company usually does not have this problem and does not need this infrastructure. It does need the same rule. A tool for an agent starts disabled, and the person responsible for it turns it on.

In a small company this rule fits into a configuration file and one name. This is my conclusion; Uber does not write about small companies. I do not know the threshold at which a central gateway pays off. I will not guess it. The signal is the moment when different teams start building the same integrations separately.

Four questions for an AI service provider

If you are buying an agent deployment, ask the provider about the layout before you ask about the model:

1. Who challenges the result of the agent's work, and is it another model, another session or a person?
2. Where is the gate the agent cannot pass without human approval, and who approves at it?
3. What number are we to beat, how will we measure it and what is its value today?
4. What happens when the number gets worse: who rolls back the change and how long does it take?

The first question checks whether a challenger exists at all. A provider who answers it with the name of a model is answering a different question.

The second question checks whether the gate stands before an irreversible operation or only after it. A report after the fact describes the damage the gate was meant to prevent.

The third question reveals whether anyone has measured the baseline. Without today's value you will not find out next quarter whether the agent improved anything. A provider who does not know the answer is selling you the impression of progress.

The fourth question is asked least often. In the claude.ai sprint the answer was flags and staged rollout, so withdrawing a visible change could come down to switching off a flag. If a provider answers that the changes are tested and there is no need to roll back, you have heard that there is no rollback plan.

The question about the model remains legitimate, but for a task within reach of several frontier models it comes fifth. It makes sense then, because you already know what role the model will fill.

These questions also work the other way round. If you are deploying agents yourself, answer them in writing before you start the first run. The answers will take one page. A missing answer to any of them is more informative than any benchmark.

What stays after the model is swapped

The post on the advisor strategy dates from April, and the claude.ai sprint took place in August on an internal model not available to the public. The model names in this essay will age faster than the essay itself.

In a company that has designed its control layout, the same things remain after the model is swapped: the thread owner, the number with a ratchet, the consent log and the rule that a tool starts disabled. These things had to be designed once. At the next model swap there is no need to design them anew, though the thresholds and baselines have to be measured again. If your deployment has to be built from scratch after you change model provider, you bought the model together with its servicing, and you have no control layout at all.

References

1. primaryWang, R., Attard, S., Issac G. (2026). How we made claude.ai 3x faster in two weeks. claude.dev blog, 23 September 2026. https://claude.dev/blog/how-we-made-claude-ai-faster/. Source of the sprint figures (p75 3.1 s → 0.55 s, geometric mean of 3.1 times across 13 measurements), of the description of roles and CI ratchets, and of the claim of more than 3,000 changes without an incident.
2. primaryAnthropic (2026). The advisor strategy: Give agents an intelligence boost. claude.com blog, 9 April 2026. https://claude.com/blog/the-advisor-strategy. Source of the results of executor–advisor pairs on BrowseComp and SWE-bench Multilingual.
3. primarySrivastava, A., Mehndiratta, D., Gill, G., et al. (2026). Designing MCP Gateway: Uber's MCP Management Platform. Uber Engineering Blog, 1 October 2026. https://www.uber.com/blog/designing-mcp-gateway/. Source of the gateway's scale and of the rule that tools stay disabled until reviewed.
4. primaryHuang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., Zhou, D. (2023). Large Language Models Cannot Self-Correct Reasoning Yet. Preprint of 3 October 2023, accepted at ICLR 2024. https://arxiv.org/abs/2310.01798
5. primaryPanickssery, A., Bowman, S. R., Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2404.13076. https://arxiv.org/abs/2404.13076
6. primaryOpenAI (2026). codex-plugin-cc, repository of the Codex plugin for Claude Code, announced on 30 March 2026 on the OpenAI Developer Community forum, https://community.openai.com/t/introducing-codex-plugin-for-claude-code/1378186. Repository: https://github.com/openai/codex-plugin-cc
7. primaryChase AI. claudex-loop, repository under the MIT licence, as of 16 October 2026. https://github.com/chaseai-yt/claudex-loop
8. primaryThe author's guard-script log, 25 September – 16 October 2026. Own material, unpublished. Source of the figures on blocks and consents used.
9. secondaryIFAC. Exploring the IESBA Code, installment 3 on the conceptual framework and evaluating threats, 2022 version according to the file address. Overview of the conceptual framework of the IESBA Code (part 1, section 120) with five categories of threats, including the self-review threat. https://www.ifac.org/system/files/publications/files/exploring-the-iesba-code-3-the-conceptual-framework-evaluating-threats-2022-version.pdf