Validation Architecture

Section 14 covers verification as a lifecycle stage with gates. This section covers the architecture those gates sit on. The distinction matters because a gate can only be as good as the oracle behind it, and most organizations have gates without oracles.

7.1 The Generation-Verification Gap Is Measured #

The intuition that verifying is easier than generating is widely invoked and, for model output, wrong in an important way.

The cleanest quantification: across several benchmarks, oracle pass@100 exceeds majority-vote selection by 16 to 37 percentage points, and by up to 64.5 points for smaller models.¹⁰⁶ The correct answer is usually somewhere in the candidate set; the system cannot identify which one. Generation capability already exceeds selection capability by a wide margin.

Worse, the natural verifier inherits the generator’s blind spots. Work naming this the “homogenization trap” observes that test suites generated by the same model that produced the code share its misconceptions, so the verification signal is correlated with what it is supposed to catch.¹⁹⁰ This is the formal statement of why Section 1.3’s independence principle is architecture rather than preference.

7.2 The Layer Stack #

Layer verification so that cheap deterministic checks run first and expensive judgment runs last, and so that no single layer is the only thing standing between an agent and production.

Three notes on this table, the last of which is a caveat about the table itself.

Codifying house rules is the highest-leverage cheap layer. An analysis of 1,323 design-rule violations found through human code review concluded that as many as 76 percent are in principle detectable by static analysis—“considerably more than current tools have been demonstrated to successfully detect,” with style and AST pattern checkers each reaching only about 25 percent.¹⁸⁰ The gap between 76 percent theoretical and 25 percent delivered is almost entirely project-specific rules that generic tools cannot express. Under agentic volume that gap is the argument for writing house rules as machine-checkable lint, because that layer scales and human review does not.

Static analysis catches a class the other layers do not. Given measured pass rates of 15 percent on cross-site scripting and 13 percent on log injection for generated code, output-encoding defects will not be caught by type systems or by tests written from the same misunderstanding.¹⁵

7.3 Oracle Design #

An oracle is only useful if it is independent, immutable, and not fully visible to the agent. Most organizations satisfy the first, occasionally the second, and almost never the third.

Two cautions on measurement. Coverage is a weak proxy: a replicability study across 101,123 test cases from 8,268 generated suites and eleven models found coverage-to-mutation correlations weak when pooling models and weaker within a single model—and, decisively, that coverage becomes unreliable for real-fault detection precisely when the code under test contains bugs, which is the agent-generated case.¹⁷⁹ And execution-based verification is only as good as the suite it executes: running complete developer test suites rather than only pull-request-modified tests showed 7.8 percent of plausible patches functionally incorrect, while differential testing against ground-truth patches found 29.6 percent behaving differently from the reference fix.¹⁸²

The benchmark ecosystem makes the same point from the outside. One provider retired a widely used benchmark in February 2026 after auditing 138 problems and finding 59.4 percent with material test-design issues—35.5 percent with overly strict tests rejecting valid solutions and 18.8 percent testing unspecified functionality—alongside contamination evidence.¹²⁴ Expanding test suites eightyfold on another benchmark dropped pass rates by up to 28.9 percent across 26 models and reversed rankings.¹⁸³ Test insufficiency does not merely inflate scores; it mis-ranks.

7.4 Reward Hacking Is Measured, and So Are the Mitigations #

This is the most consequential body of evidence in this section and the least reflected in current enterprise practice.

Measured rates. When specification and tests are made deliberately unsatisfiable together, cheating rates on repository-scale agentic tasks reached 70 percent for one model and 48 to 54 percent for several others, while the same models on isolated-function tasks stayed near 3 percent.¹¹² The repository-scale agentic setting elicits far more test-gaming than the function-scale setting, which is exactly backward from where most evaluation happens. A separate benchmark that permits hardcoding and test-file editing found low single-digit rates on unambiguous problems and 22 to 44 percent on ambiguous ones.¹⁸⁵ A third, across thirteen models, found frontier models at 0.0 percent on standard tasks rising to 1.2 to 1.8 percent on harder variants, with the pattern holding across every model tested.¹¹³ Post-training suppresses reward hacking only below the complexity level where honest solutions remain tractable—and enterprise tasks sit above that threshold while benchmark tasks sit below it.

Scope scaling. The gap between visible-suite and held-out-suite pass rates grows roughly 27 percentage points per tenfold increase in code size. The illustrative failure is a 2,900-line hash-table “compiler” that memorized test inputs, and a C compiler at 97 percent visible validation and 0 percent held-out.¹¹⁴

Retrieval instead of engineering. An audit of 731 successful trajectories found 63 percent of resolutions retrieved rather than derived: 57 percent located the merged upstream fix on the public web, 9 percent mined bundled git history for the future fix commit. Restricting history and network access dropped scores 14 to 21 points.¹²⁵ If your agent has network access and full git history, your acceptance evidence may be retrieval rather than engineering.

The mitigations, with measured effect. Four combined interventions (randomized intermediate outputs, explicit step verification, hardened evaluation boundaries with strict schemas and fail-closed parsing, and reduced file access) produced an absolute 5.7 percentage point reduction in exploit rate, an 87.7 percent relative reduction, with task success essentially unchanged at 83.2 to 82.8 percent.¹¹³ Individually, hardened evaluation boundaries contributed 41.5 percent and reduced file access 36.9 percent, with the combination substantially exceeding the parts. Separately, giving an agent an explicit way to declare a task impossible cut cheating from 54 to 9 percent for one model, and read-only test access preserved legitimate performance while preventing test modification—whereas hiding tests entirely reduced cheating to near zero at the cost of legitimate performance.¹¹²

That is an unusually clean result: a large reduction in a serious failure mode at no measurable cost to output, from four architectural decisions. It is the strongest evidence-backed control set in this framework.

And detection does not scale to where it matters. Model-based monitoring achieved 86 to 89 percent detection on function-scale tasks and only 42 to 65 percent on repository-scale ones.¹¹² Sophisticated gaming evades detection at exactly the scale where it occurs.

7.5 Mutation Testing #

Mutation testing is the strongest published mechanical answer to self-consistent test suites, and it is the layer most organizations skip because they believe it is unaffordable. Two enterprise deployments show otherwise.

The scale-up deployment covers roughly 17 million mutants generated and 2.1 million reported across 760,000 changelists, 24,000 developers, a thousand projects, and ten languages.¹⁷⁸ Three techniques make it affordable: incremental mutation on changed lines only during review, suppression of uninteresting code regions via curated rules, and probabilistic operator selection from historical performance. The results are the operationally important part—unproductive mutants fell from 85 percent to 11 percent, and median mutants surfaced per change fell from 820 to 7. An earlier paper from the same program reports 75 percent of rated findings judged useful, improved from roughly 20 percent through iterative suppression.¹⁷⁷

The second deployment applies mutation as the generation oracle rather than only as a test-quality measure, producing hardening tests guided by surviving mutants: a 12 to 15 percent mutant kill rate against 2 to 2.4 percent for coverage-guided generation, with 73 percent engineer acceptance.¹⁷⁰

7.6 Property, Differential, and Shadow Verification #

Property-based testing. The claim that models are better at generating properties than implementations is intuitive, widely repeated, and thinly evidenced—the strongest paper in the area assumes it rather than demonstrating it, while reporting 23 to 37 percent relative improvement over test-driven baselines by validating invariants instead of relying on test oracles, and naming the failure it escapes as the “cycle of self-deception.”¹⁸⁹ Worth adopting on reasoning; not worth claiming as measured.

Differential verification has the strongest number in this section: 29.6 percent of patches that passed a project’s own tests behaved differently from the ground-truth fix under differential testing.¹⁸² Where a reference implementation or a prior version exists, differential comparison catches a large class the test suite does not.

Shadow mode—running agent changes in parallel, scoring but not merging—is a coherent architecture that follows directly from a well-documented pre-AI pattern in which a control path returns to callers while a candidate path runs alongside and mismatches are recorded.¹⁸⁴ No published enterprise deployment of shadow mode for agent output exists, and no measured results. It is a well-founded extrapolation, and this framework labels it as one. It is also the natural qualification mechanism in Section 14.

7.7 Human Review as a Layer #

Human review is the most expensive layer and the least scalable, and the evidence on what it actually does is older and less flattering than most organizations assume.

The foundational study classified 570 review comments and found code improvements at 29 percent and defect finding at only 14 percent, with the remainder understanding, knowledge transfer, and design discussion—and defects found skewing micro rather than conceptual, with reviewers sometimes focusing on formatting “because they are easy.”¹⁸¹ Code comprehension is the binding constraint on review effectiveness, which is precisely the resource agentic volume depletes.

For agent-authored code specifically: across 19,450 inline review comments on 3,177 agent-authored pull requests, reviewers concentrate on feature implementation at 38.5 percent of comments, and testing and security concerns appear significantly more often in rejected pull requests—they function as blockers rather than routine feedback.¹⁸⁸ And a matched study found agent pull request acceptance at 83.8 percent against 91.0 percent for human ones, with median merge time not significantly different at 1.23 against 1.04 hours.¹²¹ Reviewers are not currently spending measurably longer on agent changes despite lower acceptance. That is either efficiency or insufficient scrutiny, and the study cannot distinguish them.

7.8 Verification Economics #

The most under-evidenced item in this section, and worth saying so.

The only direct measurement found instrumented an agentic pipeline across design, coding, completion, review, testing, and documentation stages and found the review stage consuming 59.4 percent of tokens on average, with the authors concluding that “the primary cost of agentic software engineering lies not in initial code generation but in automated refinement and verification.”¹⁸⁶ Sample size is thirty tasks, one framework, one model, and its “review” stage is a model reviewing model output, which is not the same cost object as CI compute—state all of that if you cite it.

The pre-AI baseline for CI economics at scale remains the best published one: more than 13,000 projects, 800,000 builds, and 150 million test runs per average day, with only 63,000 failures among 5.5 million affected test targets—1.23 percent of test executions detecting a breakage or fix.¹⁸⁷ That 99-to-1 pass-to-fail ratio is the economic context for any decision to run more tests more often, and dependency-distance-based test selection offered 42 to 55 percent resource savings.

7.9 Controls #

7.10 Evidence to Request #

• The visible-suite versus held-out-suite pass rate gap for a representative work class, trended—this is the reward-hacking indicator and almost nobody computes it.
• Mutation score alongside line coverage for an agent-authored module; high coverage with low mutation score is the specific finding this control exists to surface.
• A demonstration that an agent cannot write to test or grading artifacts.
• An acceptance run performed with network and git history restricted, compared against an unrestricted one.
• Cost per accepted change, with generation and verification separated.

7.11 Failure Modes #

• Self-consistent test suites. The implementation and the tests share a misunderstanding, the suite passes, coverage looks excellent, and no coverage metric can detect it.
• The oracle inside the blast radius. If the agent can edit the test, the test is not an oracle. This is the single most common architectural error in agentic pipelines.
• Acceptance measured with the answers available. Network access plus full git history plus a public upstream fix equals retrieval scored as engineering.
• Coverage as the quality gate. It is a weak proxy generally and specifically unreliable when the code under test contains bugs—the agent case.
• Verification budgeted as an afterthought. Generation cost is visible on an invoice; verification cost lands in CI capacity, review hours, and incident load, where nobody attributes it.