Flaky-Test Repair as a Deployed Work Class

35.1 Context #

The codebase is Uber’s Go monorepo: approximately 100 million lines across more than 100 projects, worked by over 6,000 engineers on more than 100 teams. The baseline problem is well characterized and predates agents by a decade. At Google, roughly 1.5 percent of all test runs report a flaky result, almost 16 percent of tests exhibit some flakiness, and about 84 percent of observed pass-to-fail transitions involve a flaky test.¹⁶² A flaky suite does not merely waste time; it destroys the signal that any delegation strategy depends on.

35.2 The Mechanism #

The system is wired into the existing ticketing system and delivers fixes to developers daily without human initiation. Its contribution is not the repair prompt. It is context selection.

Compile-time instrumentation builds a dynamic call graph from actual test execution, which contains roughly 40 percent of the nodes a static call graph would. A model-guided breadth-first traversal then expands only the paths judged relevant, on the observation that the information needed to explain a flake usually sits in leaf nodes deep in the graph rather than near the test. The authors name the problem they are solving directly: prior approaches fail in industrial settings by supplying “either too little context (missing critical production code) or too much context (overwhelming the LLM with irrelevant information).“¹⁶¹

Reproduction is brute force rather than clever. The test is executed a thousand times with Go’s race detector enabled to perturb scheduling; if it never fails, the entire test target is executed a thousand times and the results filtered. The repair budget is three context-collection attempts, two reasoning paths, and three candidate fixes per path—eighteen attempts per test, under a two-hour cap.

35.3 The Oracle #

Build validation, then test validation: the repaired test is rerun the same number of times used for reproduction, and every run must pass. Any error or timeout reverts the candidate and advances to the next.¹⁶¹

This is a statistical oracle, and it certifies exactly one property: the flakiness stopped. It cannot certify that the test still tests what it tested. That gap is not hypothetical, and the rejection analysis in Section 35.5 is what happens when it opens.

The system prompt constrains the agent to modify test code only, never production code. That constraint is the reason the oracle holds at all—an agent permitted to edit the code under test could make any test deterministic.

35.4 What Was Measured #

The end-to-end figure is arithmetic, not a reported result; the paper states the three conditional rates and does not multiply them.¹⁶¹ It should nevertheless be the planning number. An organization that budgets against 51.8 percent will under-provision by roughly a factor of three.

There is a second gap worth naming. In a controlled evaluation on 295 reproducible tests at a single commit, the system repaired 65.76 percent, against 45.42 percent for the strongest prior tool. In six months of live deployment the same metric was 47.6 percent. That is an eighteen-point benchmark-to-production drop on the same measure, before any human acceptance filter is applied. The paper reports both figures and does not draw the comparison. It is available and it is the single most useful number in the paper for anyone sizing a pilot from published results.

Root causes among accepted fixes: scheduling randomness 37 percent, unordered collection iteration 33 percent, timestamp discrepancy 12 percent, state pollution 8 percent, time dependence 7 percent, other 3 percent.

Mean repair time was 1,978.73 seconds per test—roughly thirty-three minutes—which made this the slowest of the three systems compared, notwithstanding its higher yield. Token counts and monetary cost are not reported.

35.5 What Failed #

Of 380 produced fixes, 183 were rejected. The stated reasons, in the authors’ order of importance:

Reasons two and three are the ones that matter for control design. Both describe fixes that passed the automated oracle and were still wrong. Only human review caught them. A deployment of this work class that treats oracle passage as sufficient will land assertion removal at scale, and the resulting suite will be green and worthless.

A developer survey returned nineteen responses. All nineteen found the root-cause explanations useful; mean fix-quality rating was 4.42 of 5. The most common time-saving estimate, at 42.1 percent of respondents, was less than one day per test—which is a useful corrective to the way such systems are usually sold.

35.6 What Transfers #

The reproduction requirement transfers first and hardest: 28.4 percent of reported flaky tests never reproduced, and nothing downstream can help them. Any organization planning this work class should measure its own reproduction rate before anything else, because that number caps everything.

The guardrails transfer next, and they follow directly from the rejection analysis. Assertion-count comparison and mutation score on the repaired test, both computed automatically and both gating, are the minimum defense against a fix that makes a test green by making it empty. A green build is not evidence here; it is the thing the failure mode produces.

Last, the custom-API rejection reason is a context-substrate problem, not a model problem. The organization’s own testing conventions were invisible to the system, and roughly a fifth of its rejections followed. Section 28 is the remedy.

35.7 What This Rests On #

The funnel, the oracle description, the rejection reasons, and the root-cause distribution are all reported in a peer-reviewed paper. The 17.7 percent end-to-end figure and the eighteen-point benchmark-to-production gap are derived by this framework from figures the paper reports separately. The per-reason counts behind the rejection analysis are not published, so the claim that assertion removal is common rests on the authors’ qualitative ranking rather than on a count.