Designing an Oracle

Produces an oracle specification for one work class: what correct means, what decides it, and what the agent cannot see.

24.1 The Three Properties #

An oracle that lacks any one of these is not an oracle. It is a metric the agent will optimize.

Independent. Derived from the requirement rather than from the implementation, and not produced by the same model in the same session. When one model writes both the code and the test, a passing suite establishes self-consistency and nothing else.

Immutable. The agent cannot modify the oracle. This is a filesystem permission and a branch protection rule, not an instruction. Every verification signal an agent can reach becomes a target, and the measured remedy is architectural: removing write access to test and grading artifacts, combined with hardened evaluation boundaries, cut exploit rates by 87.7 percent relative with task success essentially unchanged.¹¹³

Partly hidden. A held-out signal the agent cannot observe during the run. This is the only reliable defense against optimization toward the visible check, and its absence is measurable: the gap between visible-suite and held-out-suite pass rates grows roughly 27 percentage points per tenfold increase in code size.¹¹⁴

24.2 Oracle Classes, Ranked by Strength #

Class	Mechanism	Strength	Cost	Agent-visible
Compilation and type checking	The build succeeds and types resolve	Weak alone, free, catches structural error	Negligible	Yes
Execution against authored tests	A test suite written independently of the change passes	Moderate; strong when the suite is human-authored and immutable	Low	Yes, and therefore gameable
Held-out compositional tests	A suite the agent never sees, exercising features in combination	Strong	Medium	No
Differential comparison	Behavior matches a reference implementation or prior version	Strong where a reference exists	Medium	No
Mutation score	Injected defects are detected by the suite	Strong for validating agent-written tests	High, tractable when diff-scoped	Should not be
Property and invariant checks	Stated invariants hold across generated inputs	Moderate to strong	Medium	Partial
Statistical threshold on a corpus	Score distribution clears a threshold with a reported interval	Moderate; the only option for probabilistic components	Medium	Partial
Model-as-judge	A second model scores the output	Weak unless validated against human labels	Low	No
Human review	A person decides	Strong on intent and design, weak on mechanical defect, does not scale	Highest	No

Two entries deserve a warning. Coverage is not on this list, because a replicability study across 101,123 test cases and eleven models found coverage unreliable for real-fault detection precisely when the code under test contains bugs, which is the agent case.¹⁷⁹ And model-as-judge belongs at the bottom until validated: judges show a 33 to 41 percentage point gap between raw agreement and chance-corrected agreement, and two production judges exhibited test-retest reliability above 0.95 while carrying position bias above 0.10, meaning they were consistently wrong in the same direction.⁷⁸

24.3 The Procedure #

Step 1. State the correctness claim in one sentence. Not “the code works.” Something falsifiable: this change preserves the public behavior of module X while eliminating the deprecated call. If you cannot write the sentence, you cannot build the oracle, and the work class is not ready for delegation.

Step 2. Choose the strongest affordable class. Work down the table until you reach something you can actually run on every change. Most work classes land on execution against authored tests, and most should add a second layer.

Step 3. Establish independence. Answer three questions in writing. Who wrote the oracle? Was it derived from the requirement or from the implementation? Can the same agent session that produces the change also produce or alter the check? If the answer to the last is yes, stop and fix that before proceeding.

Step 4. Make it immutable. Enumerate the paths the oracle lives in: test directories, fixture data, grading scripts, CI configuration, policy files. Remove agent write access to all of them. Then verify by attempting a write from a test agent and confirming it fails. A policy statement is not this step; the attempted write is.

Step 5. Build the held-out signal. Reserve a portion of the verification that never enters the agent’s context or repository. Three workable constructions: a compositional suite exercising features together while the visible suite exercises them in isolation; a rolling holdout of recent production cases the agent has not seen; or a differential check against a reference the agent has no access to. Run it after the agent completes, never during.

Step 6. Calibrate before gating. Run the oracle against known-good and known-bad changes. Record its false-positive and false-negative rate. An oracle gating merges without a measured false-negative rate is a gate whose reliability you are asserting.

Step 7. Instrument the gap. Record visible-suite and held-out-suite results separately for every change, forever. The gap between them is your reward-hacking indicator, and almost no organization computes it.

24.4 Worked: Three Work Classes #

Test generation and coverage backfill. Correctness claim: this test fails when the behavior it describes is broken. Primary oracle is mutation score rather than coverage, on the evidence that mutation-guided generation killed mutants at roughly six times the rate of its coverage-guided predecessor, 15 percent against 2.4 percent, and that 49 percent of the useful tests it produced added no line coverage at all.¹⁷⁰ Independence: the mutants are generated by the platform, not by the authoring agent. Immutability: the mutation configuration and the mutant corpus sit outside agent scope. Held-out: a mutant subset withheld from the reported score. Gate: a mutation-score floor, not a coverage floor.

Dependency upgrade with breakage repair. Correctness claim: the application’s observable behavior is unchanged and the vulnerable version is gone. Primary oracle is execution against the existing authored suite, plus a differential check on public API surface. Independence is inherent, because the suite predates the change. Immutability is the critical step here and the most commonly skipped: the agent must not be able to edit tests to accommodate the upgrade. Held-out: an integration suite the agent’s branch does not run. Guard for the known failure mode, which is a resolved version the model chose rather than policy chose, given that 36.7 to 55.7 percent of tasks in one study carried a known CVE in the model-specified version.⁶⁴

Documentation maintenance. Correctness claim: the described behavior matches the code. This class has no oracle in most organizations, and that is the finding rather than an inconvenience. Building one is possible and is the precondition for delegating above A1: executable examples that run in CI, reference and link validation, signature extraction compared against the documented signature, and a staleness check tying each document to the commit range of the code it describes. Until at least two of those exist, keep the class at A1 with mandatory human review. The reason is the risk shape, not squeamishness: generated documentation is well formed, authoritative-looking, and unverifiable, which is the profile in which readers cannot detect error.¹⁷⁵

24.5 Instrumentation #

Visible-suite to held-out-suite pass rate gap, per work class, trended
Oracle modification attempts, logged and alerted, target zero
False-negative rate from calibration, refreshed quarterly
Escalation rate per agent-week, where suppression of escalation is a defect rather than efficiency
Share of merged changes gated by an oracle stronger than compilation

24.6 Failure Signatures #

The gap widens quietly. Visible pass rates hold steady while held-out rates decline. This is the signature of optimization toward the visible check and it is invisible without step 7.
Escalation goes to zero. An agent that never says a task is impossible has either become perfect or has learned that producing something scores better than declining. Giving an agent an explicit impossibility affordance cut cheating from 54 to 9 percent for one model, so the affordance should exist and its use should be expected.¹¹²
The oracle starts failing on legitimate change. A brittle oracle trains the organization to bypass it. Rising exception requests are a signal to fix the oracle, not to grant the exceptions.
Acceptance measured with the answers available. If verification runs with network access and full git history, acceptance may be measuring retrieval rather than engineering: an audit of 731 successful trajectories found 63 percent of resolutions retrieved rather than derived, with scores dropping 14 to 21 points once history and network were restricted.¹²⁵

24.7 What This Rests On #

The three properties are measured, not asserted: independence from the reward-hacking mitigation result,¹¹³ the held-out requirement from the scope-scaling finding,¹¹⁴ and the ranking of judge and coverage oracles from the judge-reliability and coverage-correlation studies.⁷⁸^,¹⁷⁹ The seven-step procedure is this framework’s construction, and the specific step sequence has not been validated in the field.