The New Bottleneck: Why Human in the Loop Doesn’t Scale

Ask how we keep AI output under control and the answer comes almost by reflex: human in the loop. A person looks at it, a person signs off, a person carries the responsibility. It sounds sensible, and in many cases it is.

My thesis is still this: human in the loop is not a solution, it is a transitional strategy. It only works as long as AI systems produce little. That assumption is becoming false right now.

The reason is simple. The volume of AI output is growing exponentially; human review capacity is not. Routing every output through a person doesn’t remove the constraint, it moves it: away from generation and onto review. What we end up with is not more control, but a new bottleneck.

For decades a quiet assumption held: humans produce work, humans review work, and both happen at the same pace. One engineer designs a plant, a second checks the design. One developer writes code, a colleague reads it. Review was never the constraint, because production and review ran at the same speed.

AI breaks that balance. Production now costs almost nothing; review costs exactly what it did before. An agent writes as much code in a night as a team did in a month. A model generates a hundred variants of a specification before the person responsible has finished their coffee.

On the left, the old picture: input and output are the same size, and the human in between keeps up. On the right, the new one: input grows, the human stays the same, and output is no more than what fits through that person. In this model, more AI brings no additional value, only a longer queue in front of the review step.

Software development is the field where AI has advanced furthest, so it shows first what is coming for everyone else. In October 2026 the Claude Code team at Anthropic described how they deal with the problem (How the Claude Code team does code review).

The numbers make the dilemma tangible. In Anthropic’s monorepo, changes landing per day went from about 2,500 in May to more than 30,000 in September 2026. Roughly 90 percent of Claude Code’s own code is written by Claude. Nobody can read 30,000 changes a day line by line. Human in the loop in the classic sense is no longer possible here, no matter how many reviewers you hire.

The team’s answer was not to remove review. They changed the question. Instead of “did a human read the code?” they ask: “what could go wrong with this change, and which check, automated or human, is the right one to catch it?” That produced a multi-layered system:

• Verification before the pull request. The agent runs tests, build and linter itself and has to show the results rather than claim them. A second agent reviews the diff in fresh context.
• AI review on every pull request. Several agents hunt for bugs in parallel; a second group votes on each finding. Only findings with enough agreement get posted. Every finding has to be answered: fix it or close it with a reason.
• Risk-based approval. A classifier reads the change and assigns a tier: no human approval, one person or two. Docs and additive features behind a flag no longer need a person; billing, the auto-updater and CI workflows always do.
• Fast detection, fast rollback. Feature flags, staged releases and internal use before public rollout. Mistakes that get through are undone in minutes.
• Incidents become the test bench. Every incident becomes a test case the AI reviewer runs against daily. The reviewer gets better with every mistake.

The result: from June to September 2026, merged pull requests more than doubled while the share that had to be reverted fell by over 40 percent. More output, fewer errors, less human time per change.

The human hasn’t disappeared. People decide whether something gets built, review architecture and risky code, read the evidence instead of the code, and own the outcome. Whoever directs an agent counts as a reviewer of the change. That is the crucial shift in perspective: the human is no longer the filter everything has to pass through, but the one who designs the review system and steps in where it pays off.

In pharmaceutical manufacturing, human in the loop is becoming a regulatory expectation. The draft of EU GMP Annex 22 “Artificial Intelligence” was released for consultation on 7 July 2025, and the comment period closed on 7 October 2025. According to its Inspectors Working Group work plan, EMA is targeting the fourth quarter of 2026 for the final text (Scilife, July 2026).

The draft is clear in its stance. In critical GMP applications, meaning anywhere patient safety, product quality or data integrity are directly affected, only static, deterministic, predictive models should be used. Generative AI and LLMs are excluded there. In non-critical applications they are allowed, but only if qualified personnel check the outputs for fitness and carry the responsibility. Where a model informs a human decision and testing effort has been reduced on that basis, the draft requires documented reviews, depending on criticality even for every single output.

The regulators’ caution is understandable. In April 2026 the FDA, for the first time, gave “inappropriate use of artificial intelligence” its own section in a warning letter: a manufacturer had let AI agents generate specifications, procedures and master production records without adequate review by the Quality Unit. Exactly the scenario Annex 22 is meant to prevent.

Still, I see two problems.

The first is the scaling problem from above. When AI in quality departments produces deviation reports, change requests, risk assessments and SOP drafts, the number of documents that need a signature grows. The number of QA staff does not grow with it. The consequence is predictable: either innovation is slowed because the AI isn’t deployed at all, or review degenerates into a formality. The Scilife article names the mechanism itself: when results arrive quickly and at scale, rubber-stamping becomes the path of least resistance. A signature alone is not evidence of a meaningful review.

The second problem is automation bias. The more reliably a system performs, the less closely the human looks. After the hundredth correct output, the hundred-and-first is no longer read, just waved through. Human in the loop then produces exactly the false sense of security it was meant to prevent: there is a signature, but no control.

To be fair: Annex 22 does not require the Quality Unit to approve every AI output. The draft is risk-based, and at its workshop at the end of June 2026 EMA explicitly discussed whether acceptability should depend on the technology or on demonstrated risk control across the lifecycle. In other words: should a method be ruled out because there is an LLM behind it, or should it be permitted once the manufacturer can demonstrate that the risks are under control for the entire period of use, from validation through to ongoing monitoring? That is the right question. Its answer will decide whether the pharma industry gets to use AI or merely administer it.

The solution is not less control, but a different architecture of control. The human moves out of the data stream and above it: designing the review system, monitoring it, and stepping in where the risk is high or the decision is irreversible. In English, that is “human on the loop” rather than “human in the loop”. Five principles make it concrete.

1. Layers, not a gate. A single review step at the end doesn’t scale because it has to see everything. Several layers that each catch part of the errors do scale, because each layer can be automated: self-verification by the producing system, independent AI reviewers with fresh context, a vote among several reviewers on each finding, a security pass. Few errors make it through every layer. For the rest there is layer five: fast detection and rollback.

2. Risk sets the review depth, not the origin. Not every output is equally dangerous. A classifier assigns each output to a risk tier, and only the high tiers reach a human. Quality assurance has known this principle for decades: no incoming-goods inspection opens every box; it works with statistical sampling plans based on AQL. Demanding 100 percent inspection for AI output is a step backwards from our own methodology.

In a GMP environment the obvious objection applies immediately: almost everything there is critical, and a static classifier would send almost everything to a human. The bottleneck would only have moved. The way out is to treat risk dynamically. Risk is a property of a class of decisions, not of each individual instance. Once a human has reviewed a class of cases several times and documented the outcome, that is evidence: the review becomes a rule for the automated reviewer and a test case for its validation. The class then counts as proven and no longer needs a human, as long as monitoring shows no deviation. ICH Q9 argues exactly this way: what matters is not the severity of a possible error alone, but severity times probability times detectability. Every review the system takes over raises detectability. The share that lands with a human therefore shrinks over time instead of growing with output.

3. Validate the process, not every output. The pharma industry already has a model for this: Real Time Release Testing under Annex 17. Instead of testing every batch at the end, the process is understood and monitored well enough that release can rest on process data. The same transfers to review systems for AI. You validate the AI reviewer like a model under Annex 22: defined intended use, independent test data, acceptance criteria, monitoring for drift. Every incident becomes a new test case the review system runs against regularly. The human then checks the review system periodically, not every single output.

4. Reversibility decides where the human must stay. A document draft can be withdrawn; a released batch cannot. The line for mandatory human approval should run where an error can no longer be corrected. Everything before it can be secured with small steps, flags and fast rollback. The smaller and more reversible the units, the less a human has to approve in advance.

5. Accountability is not review. The misunderstanding behind human in the loop is that accountability only exists if someone has read every line. That isn’t true. Whoever directs a system and releases its output is accountable for the result, even if they read the evidence rather than the code. The Claude Code team calls it “you land it, you own it”. Annex 22 says essentially the same: accountability stays with the regulated user and does not migrate to the model. Both are compatible with automated review once you decouple accountability from reading.

And it needs a metric that keeps the whole thing honest: human time per output as the optimisation target, error and incident rate as the guardrail. If the error rate rises, review gets stricter again. If it stays stable, automation may keep growing.

Put together, it looks like this: the automated layers absorb the bulk, the classifier sends only unproven cases to a human, and every human review flows back into the review system as a rule and a test case. On the first pass in a GMP environment, almost everything lands with the human. With every proven class, the automatic path widens and the red box shrinks. The human no longer sits in the main stream, but at the point where their judgement permanently improves the system.

Human in the loop was the right answer for a phase in which AI produced little and nobody knew how good it was. That phase is over. Anyone who still routes every output through a person gets no safety, only a queue, and eventually reviewers who sign rather than read.

The alternative is not giving up control. It is more demanding: build review systems that are themselves validated, classify risk cleanly, build reversibility into the processes, and direct human attention to where it is irreplaceable. Software development already shows that this works, with more output and fewer errors.

For regulated industries the decisive question will be whether the regulator ties accountability to reading every single output or to demonstrated control of the overall system. EMA has put that question on the table for Annex 22. How it is answered will decide whether we get to use AI in production or merely supervise it.

• How the Claude Code team does code review, claude.dev, 15 October 2026
• What Annex 22 means for AI governance in GMP-regulated environments, Scilife, 29 July 2026
• What is “Human-in-the-Loop”?, ECA GMP News