What the Cases Say Together

Eight cases, six organizations, four years of published record. They were selected for evidentiary quality rather than for agreement, and they do not agree about everything. What follows separates what they establish, what they merely suggest, and where they undercut positions this framework has taken elsewhere.

42.1 The Oracle Predicts the Outcome #

Arrange the eight by the strength of the mechanism that decided correctness, and the outcomes sort themselves.

Two readings follow, and only one of them is comfortable.

The comfortable one is that oracle strength governs achievable scale. Where a machine can decide correctness for every artifact, the work runs at population scale and humans supervise the system rather than the artifacts. Where correctness needs a person, throughput settles at whatever review capacity exists, and the well-run organizations throttle generation to match rather than discovering the ceiling by breaking their reviewers.

The uncomfortable one is that a machine-decidable oracle can be strong and still wrong in a specific direction. The flaky-test system’s oracle is unimpeachable on its own terms—a thousand consecutive passing runs is not a weak signal—and it still admitted fixes that removed assertions and relaxed mocks, because those changes satisfy the oracle exactly. Nearly half of everything that cleared it was rejected by humans. A machine-decidable oracle certifies the property it encodes and nothing adjacent to it, and the gap between “the property I encoded” and “the property I wanted” is where the residual review effort permanently lives.

42.2 Read the Funnel, Not the Headline #

Every case that reports a complete pipeline reports a headline number substantially better than its end-to-end yield, and in no case is the end-to-end figure the one that gets quoted.

There is also a benchmark-to-production gap wherever both are published. The flaky-test system scored 65.76 percent in a controlled evaluation and 47.6 percent on the same metric in live deployment—eighteen points, before any human filter. A pilot sized from published benchmark results, or from any single-stage percentage, will be sized wrong by a factor of two to three.

The practical rule is short. Demand the denominator, multiply the stages, and plan against the product.

42.3 Absorption Is the Invariant #

Six independent organizations, working in different languages on different problems with different tooling, arrived at the same constraint and said so in their own words.

Google throttled change generation deliberately to protect reviewers, and named review speed as the bottleneck.¹⁰⁷ Microsoft’s report describes asymmetric pressure between the person triggering the work and the people reviewing it, and observes that one person with a phone can outpace a team.¹⁰⁵ Spotify reports a 76 percent increase in pull request frequency and, in the same sentence, 76 percent more pull requests to review.¹⁰¹ Uber built a review system covering 90 percent of changes and engineered its entire post-processing stage around suppressing most of what it could say.¹⁰⁸ Meta built a system whose value is that it discards roughly 95 percent of what it generates.¹⁶⁹

None of these organizations set out to demonstrate the thesis in Section 1.2. They each discovered the same constraint independently and responded to it structurally. That convergence is the strongest evidence in this document, and it does not depend on any single figure being right.

The corollary is a planning discipline rather than a caution. Absorption capacity is a number an organization can measure, and generation should be provisioned against it. Spotify’s cohort gating, Google’s deliberate throttle, and Uber’s suppression classifier are three implementations of the same decision.

42.4 Preparation Beats Model Selection #

No case in Part VI attributes its improvement to a better model. Every one that reports an improvement attributes it to context, environment, or filtering.

Microsoft’s success rate moved from 38.1 to 69 percent across a documented set of environment and instruction changes: package-feed access, a written description of how the repository builds, platform constraints stated explicitly, testing conventions, and eventually a set of repository-specific skills.¹⁰⁵ The flaky-test system’s entire contribution is context selection—a dynamic call graph containing 40 percent of the nodes of a static one, traversed selectively—and it beat the strongest prior tool by more than 22 percentage points on that basis alone, using publicly available models.¹⁶¹ Spotify improved agent reliability by removing tools, and specifically by withholding open-ended search.²³⁶ Meta’s systems use one model throughout and locate all their value in the filters around it.^169,170

Two of these organizations also report where the absence of context cost them. Roughly a fifth of the flaky-test system’s rejections stem from fixes that did not use the organization’s own test-helper APIs, which the model had no way to know about.¹⁶¹ Microsoft’s agent produced unvalidated performance claims until it was given a benchmarking harness on real hardware.¹⁰⁵

An organization deciding where to spend the next quarter has an answer from the published record, and it is not the model.

42.5 Acceptance Measures the Organization #

The Meta result is the cleanest natural experiment available anywhere in this literature. Same tool, same model, same generation pipeline, two product teams: 90 percent acceptance against 56 percent. The obvious confound was checked and runs the wrong way—the team that accepted less rated the output more relevant. The authors’ explanation is team culture.¹⁷⁰

Microsoft’s data says something adjacent from a different direction. Agent pull requests that received direct human commits merged at 86.2 percent; those that did not merged at 55.1 percent.¹⁰⁵ The variable is not the agent’s output. It is whether a human treated that output as a draft to finish rather than a submission to judge.

Both findings point at the same operational conclusion. A pilot that reports one acceptance number across several teams has averaged away the dominant variable. Run it in two teams, compare, and treat a wide gap as a review-norm problem rather than a prompting problem.

42.6 Where These Cases Do Not Support the Framework #

Four honest concessions, because a case-study section that only confirms its own framework is not evidence. These are places where evidence that exists cuts against the framework; Section 23 covers the separate problem of areas where no evidence exists at all.

No case publishes a defect-escape rate. Microsoft publishes reverts—0.6 percent against a 0.8 percent human baseline—and that is the only direct quality signal in all eight cases. Nobody publishes production incidents attributable to agent-authored change, nobody publishes downstream defect density, and Google states outright that whether there is a long-term quality impact “remains to be seen.”¹⁰⁷ This framework’s absorption thesis is therefore well supported on throughput and unsupported on outcomes. It is a claim about what constrains delivery, not a demonstrated claim about what preserves quality.

The one apparent human control in the literature does not hold. The 65 percent against 51 percent review figures are the most-cited evidence that agent review adds signal, and they are cited that way because they look like a same-population comparison. Section 39.2 shows the two numbers are produced by different methods against different criteria, with the human figure’s population and sample unpublished. The honest status is suggestive, and Section 6 states it that way. That leaves this framework’s position on agent review resting on design reasoning and coverage data rather than on a demonstrated advantage over human reviewers.

Cost is almost entirely unpublished. Microsoft explicitly excludes compute and continuous-integration consumption from its analysis and says organizations should factor them in. The flaky-test paper reports thirty-three minutes of mean wall-clock per test and no token or dollar figure. Neither Meta paper reports cost at all. The only cost datum in the whole set is Google’s qualitative warning that “migrations often require touching thousands of files and the costs might quickly add up.”¹⁰⁷ Any economic model built on these cases is built on nothing.

One case succeeds while contradicting this framework’s insistence on measurement. Spotify automerges the large majority of millions of maintenance changes with no human in the loop, and publishes no failure rate, no rollback rate, and no defect-escape rate for any of it. By this document’s own standards that is an unverified control. It also appears to work, has for years, and rests on cohort gating and failure correlation rather than on per-change verification. The honest reading is that a temporal control—damage a few repositories, detect, stop—can substitute for a logical one at sufficient scale and sufficiently low blast radius per change. The framework does not model that substitution, and names it among the open problems in Section 23 rather than pretending to.

There is also a selection effect running through all eight. Successes are published by the organizations that built them; failures reach the public record only when they are spectacular or externally visible. The two failure cases here are a supply-chain compromise and an incident at a consumer-facing platform, not a quiet quality erosion inside an enterprise. The literature has no examples of the latter, and its absence is not evidence.

42.7 The Failures Are Permission Failures #

Neither failure case is a case of a model being insufficiently capable.

The supply-chain compromise succeeded because a workflow interpolated untrusted input into a shell, because a token carried more permission than it needed, and because a developer workstation holds credentials for everything the developer can reach. The agents involved were the novel instrument, not the vulnerability; they refused roughly a quarter of the time, and the more reliable barrier was one agent’s default confinement to its workspace—a permission, not a judgment.²²⁷

The data-destruction incident succeeded because a development agent had a credential path to a production database. The vendor’s own diagnosis is the correct one, and it is a statement about architecture: it “should never be possible.”²³¹

Both confirm the design principle in Section 1.3 more strongly than any success case does. Restrictions that can be implemented either as a prompt or as a permission should be implemented as a permission, and the two cases where that was not done are the two cases in this section that end badly.

42.8 The Short Version #

For a reader who needs one page of this section: