Assured Test Generation

37.1 Context #

The methodological frame the authors call Assured Offline LLM-Based Software Engineering asks two questions of any model-generated change: does it avoid regressing the properties of the original, and does it improve on the original in a verifiable, measurable way.²³⁹ Test generation is where those questions have crisp mechanical answers, which is why it was the first deployment.

The first system augments an existing test class rather than writing a new one. That single design choice supplies the non-regression guarantee for free: “TestGen-LLM simply augments an existing test class with additional test cases, retaining all existing test cases and thereby guaranteeing there will be no regression, by construction.”¹⁶⁹

37.2 The Mechanism and the Oracle #

Here the filter chain is the oracle, and it is the only case in Part VI where the generation step is treated as interchangeable and the assurance step is treated as the product.

Three sequential filters, each of which discards everything that does not pass. The candidate must build within the existing infrastructure. It must pass on first execution and on every one of five subsequent executions, or it is deemed flaky and discarded. It must increase coverage over the original class, or it is discarded.¹⁶⁹

The second system replaces the coverage filter with something stronger. A model generates a small number of concern-specific faults—mutants—in the class under test, guided by a plain-text description of the concern and a differential example of a real past fault. A second model acts as an equivalence detector, discarding mutants semantically identical to the original. A third generates a test that must fail on the mutant and pass on the correct code.¹⁷⁰ That is a genuine held-out signal in the sense of Section 24: the criterion is fault detection, and it is not something the generating agent can satisfy by writing an empty test.

The authors of the follow-up work are careful about which of their six assurances are real:

“The first four of these six assurances are assurances in the strict meaning of the word: they are verifiable (and, therefore, falsifiable) guarantees that leave no room for doubt.”

— Harman, O’Hearn, and Sengupta, Meta, 2025²⁴⁰

Buildability, non-flakiness, fault detection, and coverage reporting are machine-decidable. Relevance to the concern and conformity to house style are not. The gap between those two groups is where the interesting failure in Section 37.4 lives.

37.3 What Was Measured #

The first system’s filter funnel, measured across 86 Kotlin components with existing human-written test classes: 75 percent of test classes yielded at least one new case that built correctly, 57 percent at least one that built and passed reliably, and 25 percent at least one that built, passed, and increased line coverage.¹⁶⁹ These are cumulative class-level rates, not per-test-case pass rates, and the paper’s abstract states them more loosely than its body does.

At deployment across three test-a-thons between 29 October and December 29, 2023, the system was applied to 1,979 test classes and successfully improved 196 of them—roughly 10 percent. Of the improvements it did produce, 73 percent were accepted by developers.¹⁶⁹ A note for anyone citing this: the paper’s abstract says 11.5 percent where its body says approximately 10 percent, and 196 of 1,979 is 9.9 percent. The counts are the reliable part.

At the level of individual model attempts, the ratios are starker. Of 8,996 trials on one platform, 490 produced a surviving test; of 23,535 on another, 831 did. Roughly four to five percent of generation attempts survive the filters. The system’s value is almost entirely in what it throws away.

The second system ran from 28 October to December 31, 2024 across 10,795 Android Kotlin classes on seven platforms. It generated 31,677 mutants, of which 9,095 built and passed. Of those, 2,246 were syntactically identical to the original, 1,016 were judged equivalent, 1,173 produced no judgment, and 4,660 were usable. From those it produced 571 hardening test cases—294 of which added line coverage and 277 of which did not.¹⁷⁰

That last split is the most consequential number in either paper:

“Approximately half of all tests generated, although finding faults missed by all existing tests, do not add line coverage. Had we used coverage as our sole test adequacy criterion, the platforms would thus have continued to be vulnerable to regressions denoted by up to approximately half of the faults.”

— Foster et al., Meta, 2025

The filter you choose determines what you catch. A coverage gate would have discarded half the useful output of a system explicitly designed to find faults.

37.4 What Failed, and the Thirty-Four-Point Spread #

Overall acceptance across 191 reviewed tests was 73 percent. Disaggregated by team, on the same tool, the same model, and the same generation pipeline:

Thirty-four percentage points separate the two products. The authors checked the obvious explanation and it does not hold: WhatsApp engineers rated the tests more relevant to the privacy concern than Messenger engineers did, at 39 percent against 33 percent, and still accepted far fewer. Their explanation is the honest one:

“We believe that the different acceptance rates may simply denote different cultures between different teams; deciding to land a test into production is an inherently subjective judgment and may be influenced by team culture.”

— Foster et al., Meta, 2025

This is the strongest published evidence in Part VI that acceptance rate measures the receiving organization, not the tool. Any pilot that reports a single acceptance number across multiple teams has averaged away the variable that actually determines the result.

Two secondary findings sit alongside it. Only about 36 percent of accepted tests were judged relevant to the privacy concern that motivated them, and engineers accepted them anyway because they were useful for other reasons. And the team’s own pre-screening turned out to be wrong: ten tests were excluded in the first phase on the assumption that null-pointer-exception tests would be rejected as irrelevant, and when pre-screening was dropped in the second phase, acceptance stayed high.

The independent critique available is narrow but fair. The evaluation is Kotlin only; whitespace-sensitive languages stress the build filter differently. Yield of roughly 10 percent is the real-world ceiling, not the 75 percent that gets quoted. And the system is not autonomous—it requires human review between iterations.²⁴¹

Neither paper publishes cost, token counts, or compute.

37.5 What Transfers #

Adopt the augment-only constraint wherever it applies. Generating additions to an existing artifact rather than replacing it converts non-regression from something you have to test into something that is true by construction. It is the cheapest assurance in this document.

Replace coverage with fault detection as the gate on agent-written tests. The 294-to-277 split is the evidence, and Section 24.4 gives the procedure.

Expect and instrument the team spread. Before scaling a test-generation program, run it in two teams and compare acceptance. If the gap is wide, the intervention is review-norm alignment, not prompt engineering.

And treat generation yield as a cost line rather than a headline. Four to five percent trial survival is a compute budget, and neither paper reports what it cost.

37.6 What This Rests On #

Both papers are first-party industry-track reports without independent artifact evaluation. The funnel figures, the team acceptance split, and the coverage-versus-fault-detection result are directly reported. The internal inconsistency in the first paper’s improvement rate is unresolved in the source. The claim that acceptance measures the organization rather than the tool is this framework’s reading of the authors’ own team-culture explanation, and it rests on a single two-team comparison with roughly twelve reviewers in total.