Backlog Work at Repository Scale

34.1 Context #

dotnet/runtime is roughly nine years old as a repository and considerably older as a codebase, carrying lineage from coreclr and corefx before them. It spans managed and native code, targets multiple operating systems and processor architectures, and is maintained by a large permanent engineering staff alongside a substantial external contributor community. It is, in other words, the opposite of the environment in which agent benchmarks are run.

The reporting window runs from May 19, 2025 to March 22, 2026. Across that period the repository received 6,181 pull requests from all sources. The coding agent accounted for 878 of them, or 22.2 percent of everything Microsoft-originated.¹⁰⁵

34.2 The Mechanism #

An engineer assigns a GitHub issue to the agent. The agent provisions an ephemeral Linux environment, builds the repository, attempts the change, runs tests, and opens a pull request. Review then proceeds through the repository’s ordinary process, with one significant difference: humans may push commits directly onto the agent’s branch rather than only commenting on it.

That difference turns out to matter more than anything else in the report.

34.3 The Oracle #

There is no purpose-built oracle in this case, and its absence is instructive. Correctness is decided by the repository’s pre-existing continuous integration—a large authored test suite the agent did not write and cannot usefully modify without a reviewer noticing—followed by human code review.

This is the weakest oracle configuration that still works, and it works for a specific reason: the suite predates the agent by years, it is enormous, and the review culture around it is unusually strong. An organization with a thin suite and a permissive review norm cannot reproduce this result by adopting the same tool. The tool is not what is doing the work.

34.4 What Was Measured #

Across seven repositories the same agent produced 2,963 pull requests with 1,885 merged, a rate of 68.6 percent.¹⁰⁵

Success varied sharply by task type: removal and cleanup 84.7 percent, testing 75.6 percent, refactoring 69.7 percent, bug fixes 69.4 percent, documentation 68.1 percent, upgrades 67.4 percent, features 64.5 percent, and performance work 54.5 percent. The ordering is the finding. Deleting code and writing tests beat writing features by roughly twenty points, and performance work—which requires reasoning about a machine the agent cannot measure—sits at the bottom of the legitimate categories.

Three further measurements deserve to be carried into any planning conversation.

Human commits into the agent’s branch. Of 878 agent pull requests, 396 received direct human commits and 280 of those merged, a rate of 86.2 percent. The 482 that received none merged at 55.1 percent. The report’s own phrasing is that success “jumps to 86%. Without that intervention, it’s 55%.“¹⁰⁵ Collaboration outperformed supervision by thirty-one points.

Review cost. A merged agent pull request drew 16.5 comments on average against 12.4 for a merged human one, with medians of 10 and 7. Heavy iteration—more than twenty comments—occurred on 18.9 percent of agent pull requests against 11.8 percent of human ones. Review was also concentrated: the top ten reviewers wrote 61 percent of all comments on agent pull requests, and the top two wrote 36 percent.

Reverts. Three of 535 merged agent pull requests were reverted, or 0.6 percent, against 33 of 4,251 merged non-agent pull requests, or 0.8 percent. This is the only direct quality signal in the report, and it does not show degradation.

The preparation effect is reported two ways, and they should not be merged. Measured by calendar, the success rate ran from 41.7 percent in the first month to approximately 71 percent across the most recent quarter. Measured by cohort, pull requests created before the first environment and instruction changes succeeded at 38.1 percent and those created after at 69 percent.¹⁰⁵ Either framing supports the same conclusion—that the change came from preparation rather than from model upgrades—but they are different comparisons and a document that quotes 41.7 to 69 percent is quoting neither.

What the preparation consisted of is enumerable, and it is mostly unglamorous. Firewall rules permitting access to the package feeds the build requires. A repository instruction file describing structure and build process. Guidance to use targeted rather than full builds. Documentation of the fact that the agent runs on Linux only, which is a hard constraint in a codebase with substantial Windows-specific implementation. Testing conventions, including the prohibition on asserting exact exception messages in a localized product. Architectural boundaries. Later, separate instruction files per agent, a rebuilt setup step that cut environment provisioning from over twenty minutes to a few, and a set of eight repository-specific skills covering benchmarking, regression testing, continuous integration failure analysis, and triage.

34.5 What Failed #

Native and platform-specific code was persistently weak, for the structural reason that the agent could not compile or execute Windows-only paths from a Linux environment. Performance work produced unvalidated improvement claims until the agent was given a benchmarking harness on dedicated hardware. External contributions were out of reach entirely, because the source branch lives in a fork. Vague issues produced vague changes, and the agent did not extend beyond the scope it was handed.

Autonomy was more limited than the merge rate suggests: 62 percent of merged pull requests began with exactly two commits, meaning the agent made one attempt and then waited.

The report’s most quoted passage is about the reviewer, not the agent:

“The bottleneck has moved. AI changes the economics of code production. One person with good judgment and a phone can generate PRs faster than a team can review them. This creates asymmetric pressure: the person triggering CCA work feels productive (‘nine PRs!!’), while reviewers feel overwhelmed (‘nine PRs??’).”

— Stephen Toub, Microsoft, March 23, 2026

Two things are absent from the report and the author says so. Compute cost and continuous-integration resource consumption, including failed runs, are not analyzed. Downstream quality outcomes beyond the revert rate are not quantified.

And the earlier history is worse than the ten-month aggregate implies. In May 2025, within days of the agent being enabled, a set of its pull requests in this repository became a widely circulated example of the technology failing in public, including one implementing globalization comparison behavior that reviewers rejected for not accounting for locale-specific collation and that never merged.²⁴³ That episode sits inside the 41.7 percent first-month cohort. It is the same program.

34.6 What Transfers #

The finding with the widest application is the human-commit result, because it inverts the default operating assumption. Most organizations treat agent output as something to be judged: approve, request changes, or close. This repository’s data says the higher-yield posture is to treat it as a draft to be finished, and that the difference is worth thirty-one points of merge rate.

The second transferable finding is that the improvement came from environment and documentation work that any organization can do and most have not. Package-feed access, a written description of how the repository is built, and an honest statement of what the agent’s execution environment cannot reach are not advanced practice. They are the difference between 38 percent and 69 percent.

What does not transfer is the denominator. This repository has a mature suite, a strong review culture, and named reviewers with the standing to reject. Strip any one of those and the same tool produces the same volume with none of the assurance.

34.7 What This Rests On #

Every figure in this chapter comes from a single first-party report, and its author’s own caveat should be carried alongside them: agent pull requests “are not randomly sampled; they reflect deliberate choices about which tasks to assign to an AI agent,” and comparisons against human pull requests are “between fundamentally different populations.”¹⁰⁵ The 67.9 percent is not a capability measurement. It is the merge rate of the work someone chose to delegate.