The AI SDLC / Part V / §29
Section 29 of 44 5 min read

The Review Operating Model

Produces risk tiering, an auto-merge policy with a measured false-negative rate, and a seeded-defect probe.

Produces

A risk-tiering rule set, an auto-merge policy with a measured false-negative rate, a reviewer rotation, and a seeded-defect probe on a schedule.

Run this when

As soon as agent-authored change is a material share of volume. Section 3.3 shows review saturation arriving before organizations notice it, and the merge record looks identical whether review happened or was waived.

29.1 The Problem, Stated Numerically #

Telemetry across roughly 22,000 developers found median review time up 500 percent alongside pull requests merged without review up 31.3 percent.119 Those two together are the signature: reviewers are queuing and waiving simultaneously. A study of merged agentic pull requests found 79.1 percent of a manually inspected sample showing no observable reviewer interaction at all.31

Uniform review does not survive this. The question is not whether to triage but on what basis.

29.2 Risk-Tier the Change, Not the Author #

Reviewers already do this implicitly, taking a median 3.92 hours on security-relevant agent pull requests against 0.11 hours otherwise.171 Formalize what is already happening.

TierTriggersRequirement
R3Senior mandatoryAuthentication, authorization, cryptography, payment paths, data deletion, migrations, public interfaces, infrastructure definitions, agent and tool configuration, CI configuration, dependency manifestsNamed senior reviewer; no auto-merge under any circumstance; second reviewer for R3 changes that are also agent-authored
R2Standard reviewBusiness logic, internal interfaces, non-trivial refactoringAny qualified reviewer; oracle must pass; auto-merge prohibited
R1Light reviewMechanical change within a registered work class, bounded size, oracle strong and passingSampled human review at a defined rate; auto-merge permitted under 30.3

Tiering is enforced by path and change-shape rules in the platform, not by reviewer judgment at the moment of review. Configuration files belong in R3 and are the most commonly misfiled, because they do not look like code.

29.3 Auto-Merge, Honestly #

Auto-merge is the practice that makes Stage 3 economically different from Stage 2, and it is also where an organization can quietly stop verifying. One published enterprise practice describes it as “auto-merging what’s safe, concentrating review where it matters most,” and that is the entire published record.101

Four preconditions, all required.

1

The change is R1 inside a registered work class with a demonstrated, immutable oracle.

2

The oracle has a measured false-negative rate from Section 24 step 6. Not an assumed one.

3

A held-out signal runs post-merge, with the visible-to-held-out gap monitored.

4

Sampled human review continues at a rate high enough to detect drift, with the sample reviewed by a senior engineer and the findings fed back.

Set the sampling rate from your own defect economics, not from a published figure, because there is none. No published study measures human review effectiveness on agent-authored code against a ground-truth defect set, and none measures rubber-stamping rates in agent pull request review. Any auto-merge policy is therefore built on your own measurement or on nothing.

29.4 Collaborate on Hard Changes #

The most actionable review finding in the enterprise literature is not about review at all. When humans committed directly into an agent’s pull request rather than reviewing and returning it, success rose from 55 to 86 percent.105

That is a large effect and it argues against a pure reviewer posture on complex work. The operating rule: for R2 and R3 changes that fail once, the second attempt is collaborative rather than another review cycle. Reviewing and returning is efficient for mechanical defects and expensive for design ones.

29.5 The Seeded-Defect Probe #

Automation complacency is the same underlying phenomenon as automation bias, with attention allocation at its center rather than two separate failure modes.210 A reviewer whose attention has been reallocated by reliable automation does not become a worse reviewer; they become a slower one to notice.

The probe is the only mechanism in this framework that measures whether review is actually happening.

1

Construct defects representative of what your oracles do not catch: authorization boundary errors, output encoding in the wrong context, silent fallbacks, off-by-one in pagination, error suppression.

2

Seed them into the ordinary review stream at a low rate, indistinguishable from real changes.

3

Record catch rate, median review time, and diff size at approval for both seeded and unseeded changes.

4

Report catch rate as a control-efficacy metric, and treat a decline as a signal to reduce merge rate rather than to exhort the team.

5
Tell the team the probe exists

before you run it. Announce the practice, not the instances. A probe run covertly is an evaluation of individuals; a probe run openly is an evaluation of the system, and only the second produces honest behavior.

Rotate what reviewers see. Attention settles on the predictable, so a reviewer who always reviews the same work class will detect less within it over time. This is a reasoned application of the complacency literature to review scheduling and it is not measured in a software setting.

29.6 Push the Mechanical Down #

Human review is the most expensive layer and the evidence on what it delivers is less flattering than most organizations assume: a classification of 570 review comments found code improvements at 29 percent and defect finding at only 14 percent, with the remainder understanding, knowledge transfer, and design discussion.181

Meanwhile up to 76 percent of what review catches is in principle detectable by static analysis, against roughly 25 percent that generic tools actually deliver, with the shortfall being project-specific rules no generic tool can express.180 That gap is the work queue. Every house rule you codify as machine-checkable lint is review capacity returned.

Use human review for design, architecture, intent, and whether the change should exist at all. Those are the things it is measurably good at and the things nothing else does.

29.7 Instrumentation #

  • Absorption ratio: review hours required against senior engineering hours available
  • Share of changes merged without human review, by tier and by authorship class
  • Median and 95th-percentile review duration, and diff size at approval
  • Seeded-defect catch rate and median review time on seeded changes
  • Auto-merge false-negative rate from post-merge held-out results
  • Share of house rules codified as automated checks

29.8 Failure Signatures #

  • Approval rate near 100 percent with review times in seconds. Not a control. Instrument both and act on the pair.
  • Auto-merge scope creeping. It expands one work class at a time, each individually defensible. Require the four preconditions on every expansion.
  • The probe that always passes. Either the seeded defects are too easy or they are recognizable. Rotate their shape.
  • Rising review exceptions. Every exception granted to hit a date is a permanent hole unless it carries an expiry.
  • Configuration filed as R1. The most common tiering error, and the one every major disclosed vulnerability in agentic tooling went through.

29.9 What This Rests On #

The saturation telemetry, the no-interaction merge finding, the security review-latency differential, the collaboration effect, the review-efficacy classification, the static-analysis ceiling, and the automation-complacency synthesis are measured.119,31,171,105,181,180,210 The auto-merge preconditions, the probe design, and the reviewer rotation are this framework’s construction, and the absence of any published measurement of review effectiveness against agent-authored code means every organization is calibrating this on its own data.

References cited in this section

8 of 243 · numbering matches the PDF

  1. 119Faros AI, AI Productivity Paradox research report, March 2026, reported in "More Code, More Bugs," ADTmag, April 22, 2026. 22,000 developers, 4,000+ teams, two years of telemetry. Vendor-published; figures vary across the vendor's own publications and are not a coherent series.
  2. 31Sien Reeve O. Peralta, Fumika Hoshi, Hironori Washizaki, Naoyasu Ubayashi, Inase Kondo, Yoshiki Higo, Hiroki Mukai, Norihiro Yoshida, Kazuki Kusama, Hidetake Tanaka, and Youmei Fan, "Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study," 23rd International Conference on Mining Software Repositories (MSR '26), arXiv:2605.22534. 11,048 closed agentic pull requests, 9,799 human-reviewed, 717 manually inspected.arxiv.org/abs/2605.22534 ↗
  3. 171Mohammed Latif Siddiq, Zhao, Lopes, Casey, and Santos, "Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub," Information and Software Technology, arXiv:2601.00477.arxiv.org/abs/2601.00477. ↗
  4. 101Niklas Gustavsson, "Coding Is No Longer the Constraint," Spotify Engineering, June 3, 2026. First-party.engineering.atspotify.com/2026/6/code-with-claude-coding-is-no-longer-the-constraint ↗
  5. 105Stephen Toub, "Ten Months with Copilot Coding Agent in dotnet/runtime," .NET Blog, March 23, 2026. Vendor-adjacent — Microsoft reporting on a Microsoft product — but fully denominated and unusually candid.
  6. 210Raja Parasuraman and Dietrich H. Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration," Human Factors 52, no. 3 (June 2010): 381–410, DOI 10.1177/0018720810376055.doi.org/10.1177/0018720810376055 ↗
  7. 181Alberto Bacchelli and Christian Bird, "Expectations, Outcomes, and Challenges of Modern Code Review," ICSE 2013. Still the best evidence on what review actually does versus what practitioners believe it does.
  8. 180Farima Mehrpour and Thomas D. LaToza, "Can Static Analysis Tools Find More Defects? A Qualitative Study of Design Rule Violations Found by Code Review," Empirical Software Engineering 28, no. 1 (November 2022), DOI 10.1007/s10664-022-10232-4.doi.org/10.1007/s10664-022-10232-4 ↗
PDF