Qualifying an Agent for Write Access

Produces a qualification record: what was tested, what it scored, who approved it, and when it re-qualifies.

30.1 Start by Discarding the Benchmark #

Benchmark scores are not a qualification signal, and using them as one is the most common error in this decision. The gap is measured from three directions.

Agents resolving 72.8 percent of a widely used verified benchmark resolve 18.75 to 25 percent of realistic multi-file evolution tasks drawn from release notes.¹¹⁵ One major provider retired that benchmark in February 2026, having audited 138 problems and found 59.4 percent with material test-design issues, alongside contamination evidence.¹²⁴ And an audit of 731 successful trajectories found 63 percent of resolutions retrieved rather than derived, with 57 percent locating the merged upstream fix on the public web and 9 percent mining bundled git history for the future fix commit.¹²⁵

Benchmarks measure capability on curated tasks under no organizational constraint. Qualification measures behavior under yours.

There is no published enterprise standard for agent qualification, no accepted protocol, and no agreed criterion for granting repository write access. What follows is a defensible internal protocol. Adopt it, instrument it, and describe it internally as invented rather than inherited.

30.2 The Four Stages #

Stage A — Sandbox. The agent runs in an isolated environment with no access to production credentials, no network egress beyond an allowlist, and a disposable copy of a real repository. Objective: establish that it can complete representative tasks at all, and observe how it fails. Run with network and git history restricted, so that what you measure is engineering rather than retrieval.

Stage B — Shadow. The agent receives real tasks from the real queue and produces real diffs that are scored and discarded. Humans do the work independently. You compare.

This is the highest-information stage and the one to invest in. It is also, honestly, the least evidenced: the pattern follows directly from a well-documented pre-AI practice in which a control path returns to callers while a candidate path runs alongside and mismatches are recorded,¹⁸⁴ and no published enterprise deployment of shadow mode for agent output exists. Build it with instrumentation and treat your own numbers as the evidence base.

What to score in shadow:

Would this diff have merged, adjudicated blind by a reviewer who does not know the author
Diff distance from what the human actually did, and whether differences are stylistic or semantic
Oracle result, including the held-out signal
Scope adherence: did it touch only declared paths
Cost per task at p50 and p95
Failure shape, categorized against Section 5.8

Stage C — Canary. Write access to a low-blast-radius repository, R1 changes only, every change reviewed, tier A2. Objective: observe real merge behavior, real review cost, and real conflict rate with human work in flight.

Stage D — Graduated scope. Expansion by repository, work class, or tier, one dimension at a time, each with its own evidence. Never two at once, because a regression after a double expansion is unattributable.

30.3 Criteria That Are Organizational, Not Technical #

Set thresholds from your own baseline per Section 27 step 8. The criteria below are the dimensions; the numbers are yours.

Dimension	What it measures	Why it matters
Merge rate against the human baseline in the same repository	Capability under your constraints	The only comparison that means anything
Revert rate	Whether merged work survives	The least-measured dimension in the literature; one report gives 0.6 percent for agent changes against 0.8 percent for others¹⁰⁵
Review cost per accepted change	The load it imposes	16.5 comments per merged agent change against 12.4 for human ones in one deployment¹⁰⁵
Scope adherence	Whether it stays inside its envelope	Predicts blast radius under expansion
Conflict rate	Behavior alongside other work	19.8 percent within one agent product, 41.7 percent across products³²
Oracle interaction	Attempts to modify tests, configuration, or grading	Any attempt is a qualification failure, not a data point
Escalation behavior	Whether it declines when it should	An agent that never escalates has learned that producing scores better than declining
Visible-to-held-out gap	Optimization toward the visible check	The reward-hacking indicator from Section 24

Two of these are pass-fail rather than threshold: an oracle modification attempt, and a zero escalation rate across a meaningful sample.

30.4 Re-Qualification #

On model version change, always. A provider model change is a production change to the system whether or not your code changed. Behavior can shift materially under a stable API surface: one measurement found accuracy on a specific task falling from 84 to 51 percent across three months, alongside reduced instruction-following.⁸⁷

On scope expansion, always. Verification degrades with scope rather than with model quality, and the reward-hacking gap grows roughly 27 percentage points per tenfold increase in code size.¹¹⁴ A qualification earned at module scale does not transfer to service scale.

On a calendar, at minimum quarterly. Drift accumulates in the substrate, the codebase, and the model together.

30.5 Instrumentation #

Qualification records with approver, criteria, scores, and expiry, for every A2-and-above deployment
Share of deployments operating on an expired qualification, target zero
Shadow-mode agreement rate over time, which is your leading indicator of capability change
Re-qualification pass rate on model version change

30.6 Failure Signatures #

A benchmark score in the qualification record. It measures a different thing under different conditions.
Qualification once, forever. The model changed underneath it.
Shadow mode skipped for time. It is the only stage that produces evidence without risk, and it is always the one cut.
Expansion on two dimensions at once. Any regression becomes unattributable, and the response becomes a guess.
Scores from an environment with network and full git history. You measured retrieval.

30.7 What This Rests On #

The benchmark-transfer gap, the benchmark retirement audit, the retrieval finding, the conflict rates, the revert and review-cost figures, the model drift measurement, and the scope-scaling result are measured.¹¹⁵^,¹²⁴^,¹²⁵^,³²^,¹⁰⁵^,⁸⁷^,¹¹⁴ The four-stage protocol, the criteria set, and the pass-fail rules are this framework’s construction, and shadow mode specifically has no published enterprise precedent for agent output.