Stage Five — Testing and Verification

14.1 What Changes at This Stage #

Three distinct verification problems now coexist, and they require different machinery.

Verifying generated code is the familiar problem at unfamiliar volume, with a shifted defect distribution. The measured 55 percent security pass rate and the shift toward architectural and privilege-boundary flaws mean that verification effort should be reallocated: away from the syntactic and local defects that generation has largely eliminated, toward the classes generation systematically produces.^15,36

Verifying probabilistic components is a genuinely new problem for most engineering organizations. There is no input for which output is guaranteed, so acceptance cannot be binary. The discipline that applies is statistical, and the reference treatment argues that evaluation should be understood as sampling from an unobserved super-population, which makes confidence intervals meaningful rather than decorative and makes power analysis a precondition for sizing an evaluation set.⁷⁴ A two-point difference on a two-hundred-item evaluation set is usually indistinguishable from noise, and most production evaluation suites report raw pass rates without intervals.

Verifying the agents themselves is the least developed area in this entire framework. There is no published enterprise standard for agent qualification, no accepted protocol, and no agreed criterion for when an agent may be granted repository write access. Available benchmarks measure capability on curated tasks, and benchmark saturation is now actively misleading: leading systems score above 95 percent on the widely used verified software engineering benchmark, with seven of eighty-six evaluated models at or above that mark, while real-world agentic pull request merge rates sit at 63.1 percent and the security pass rate for generated code sits at 55 percent.^75,31,15 Benchmark performance is not a qualification signal. Say so internally before someone cites it in a procurement decision.

A fourth shift cuts across all three: the test suite is now also generated. When the same model writes the implementation and the tests, passing tests establish self-consistency rather than correctness. This is the specific case where the independence principle in Section 1.3 binds hardest.

14.2 Elements to Adopt #

Independence between generation and verification. Tests for agent-authored code are derived from the requirement, not from the implementation, and preferably not by the same agent in the same session. Where an agent generates both, an independent check (mutation testing, property-based testing, coverage of specified behaviors rather than of lines, or human review of the test’s intent) establishes that the tests would fail if the implementation were wrong. Mutation testing is the most practical mechanical answer available, because it directly measures whether a test suite detects injected defects.

Evaluation harnesses as versioned, owned artifacts. For every model-backed capability, maintain an evaluation set, a metric definition, a threshold, and a measurement cadence. Own the metric definitions rather than inheriting them: “faithfulness” computes differently in each major evaluation framework, and scores are not portable between tools.⁷⁶ The transport layer is converging faster than the semantics—a point Section 16 returns to—but evaluation definitions remain an organizational artifact.

Time-windowed evaluation sets. Any evaluation corpus that has existed on the public internet for a year should be assumed present in the next model’s training data. The reference design pattern annotates problems with release dates so a model can be scored only on material published after its training cutoff.⁷⁷ The transferable enterprise practice is a rolling holdout of recent, never-published production cases, plus periodic re-derivation of the golden set from live traffic. This is also the honest answer to why evaluation scores sometimes improve without the system improving.

Validated judges, or no judges. Model-as-judge evaluation is widely deployed and widely over-trusted. A systematic evaluation of twenty-one judge models across three benchmarks, comprising roughly 541,000 individual judgments, found a 33 to 41 percentage point gap between exact-match agreement and chance-corrected agreement, meaning headline agreement figures systematically overstate judge quality; found judge rankings shifting by up to fourteen positions depending on which benchmark was used; and—the most instructive finding—found two production-deployed judges with test-retest reliability above 0.95 while exhibiting position bias above 0.10.⁷⁸ They were consistently wrong in the same direction, which is the worst failure mode precisely because it presents as stability. A judge that gates a release must itself be validated against human labels, and reliability must not be mistaken for validity.

Deterministic criteria wherever they are available. Not every acceptance criterion for a model-backed feature requires a judge. Schema conformance, required-field presence, refusal behavior, tool-call validity, latency, cost per task, and prohibited-content checks are all deterministic and should carry as much of the acceptance load as possible, with judged criteria reserved for what genuinely cannot be checked mechanically.

Security testing reallocated to the observed defect profile. Given measured pass rates of 15 percent on cross-site scripting and 13 percent on log injection, and a defect shift toward privilege escalation and architectural flaws, verification should weight output-encoding contexts, authorization boundaries, and inter-component trust relationships far more heavily than the syntactic classes.^15,36 Static analysis remains necessary and is weakest exactly where the profile is shifting, which argues for supplementing it with architectural review and authorization-path testing rather than for buying more scanners.

Adversarial testing of the delivery pipeline itself. Red team the toolchain, not only the product. Concretely: attempt prompt injection through an issue, a pull request description, a code comment, a dependency README, and a tool response, and establish whether an agent’s behavior changes. Attempt to introduce a dependency that does not exist. Attempt to modify an agent rule file with concealed characters and see whether review catches it. Attempt to have an agent approve its own change. The published red-teaming methodology worth borrowing from is scored as an attack success rate across a defined strategy set, which produces a comparable number rather than a narrative.⁷⁹ The relevant risk categories for agents specifically are prohibited actions, sensitive data leakage, and task adherence.

Agent qualification before write access. In the absence of a standard, adopt a defensible internal protocol and be explicit that it is internal. The practical sequence is: sandboxed trial in an isolated environment; shadow mode, where the agent produces diffs that are scored against what humans actually did but never merged; canary scope on a low-blast-radius repository with full review; and then graduated scope expansion with measured outcomes. The qualification criteria should be organizational—defect rate, review rework rate, scope adherence, conflict rate—rather than benchmark scores.

Non-functional verification restored to the gate. Accessibility conformance, performance budgets, and resource ceilings become pipeline gates rather than periodic audits. The population-scale accessibility regression is the strongest available evidence for what happens when they are not.³⁷

14.3 Controls #

14.4 Evidence to Request #

• The evaluation set for a model-backed capability, with the date it was last refreshed and the method by which contamination is prevented.
• The judge validation record: agreement with human labels, chance-corrected, with bias measurements.
• Mutation testing results for a representative agent-authored module—a high line-coverage figure alongside a low mutation score is the specific finding this control exists to surface.
• The most recent toolchain red team report and its attack success rate.
• The agent qualification record for the agent with the broadest current write access, including who approved it and against what criteria.
• Findings generated against findings adjudicated for the trailing quarter.

14.5 Failure Modes #

• Self-consistent test suites. The implementation and the tests share a misunderstanding, the suite passes, and coverage metrics look excellent. This is not detectable by any coverage measure and is the single most likely way a defect reaches production in a heavily generated codebase.
• Evaluation theater. An evaluation set exists, has not been updated in a year, has probably leaked into training data, and reports a number that goes up. Nobody has computed an interval. The number is reported to executives as a quality metric.
• Judges gating releases unvalidated. A model scores another model’s output, the score is treated as ground truth, and the systematic bias in the judge is invisible because the judge is consistent.
• Benchmark scores used as qualification. A near-saturated public benchmark is cited to justify granting an agent broader write access. The benchmark measures capability on curated tasks under no organizational constraint, and the merge-rate and security data say it does not transfer.
• Triage suppression. The finding volume exceeds capacity, so the threshold rises. The dashboard improves. This is the failure mode most likely to be presented internally as a success.