Two Migrations, Two Oracles

36.1 Why These Two Are Paired #

Large-scale migration is the work class where agentic engineering is most often assumed to be obviously superior, because the work is repetitive, well specified, and enormous. It is also the work class with the best published evidence on both sides of the method question, which makes it the only place in the current literature where a controlled-ish comparison is possible.

The pairing is not “AI versus no AI.” Both organizations use language models heavily and both were doing so at the time. The pairing is about where the model sits in the pipeline, and what decides whether a transformed artifact is correct.

36.2 Google: The Model Generates, Humans Decide #

Google’s published migration work covers a twelve-month effort by three developers to migrate 39 distinct identifiers from 32-bit to 64-bit across an advertising codebase exceeding 500 million lines, plus three other migration families reported separately.^107,166

The architecture divides labor explicitly. Finding the places to change is deterministic: the Kythe code index resolves direct references, then indirect references to a maximum distance of five. Categorizing them is deterministic, by regular-expression classifiers into buckets for definitely-changed, definitely-irrelevant, needs-investigation, and manual-review. Only the edit itself is model-generated, three candidate modifications per file with one chosen at random from the successful attempts.

The authors state the division plainly:

“The LLM role is focused on the edit generation. The parts where we need to identify locations at which to make changes, and where we need to validate that the right thing took place, are handled mostly using deterministic AST techniques with a few cases supported by the LLM as well.”

— Nikolov et al., Google, January 2025

The oracle is a stack of six gates, and the last one is a person. A candidate change is invalid if the model returned no completion, if it changed only whitespace, if the result fails to parse or produces an identical syntax tree, if a second model call judges the change unnecessary, if the build breaks, or if regression tests fail. Everything surviving that is then reviewed visually by a developer, repaired by hand where needed, and sent to the owning team for acceptance.¹⁶⁶

The outcome across the identifier migration: 595 changes containing 93,574 edits, of which 69.46 percent of edit distance was model-authored. At the level of whole changes, 35.97 percent landed with no human edit at all, 38.48 percent were model-generated and then human-edited, and 25.55 percent were written by hand. Review surface was 306 reviewers across 149 teams, 37 offices, and 12 time zones. The prior manual attempt at a single identifier had taken around two years.¹⁶⁶

Two definitional points govern how these numbers may be used. First, success in this program is throughput, not quality: “for each migration described we have defined success as AI saving at least 50% of the time for the end-to-end work,” including finding sites, reviewing, and rolling out.¹⁰⁷ Second, the 50 percent is self-reported. The paper’s own threat-to-validity section states that it “is based on developer perception and estimations, and not on precise time tracking data.”¹⁶⁶

And the constraint that binds is named without hedging:

“The bottleneck in the process was the speed at which engineers could review the changes. We purposefully limited the number of changes we generate every weak [sic] to avoid overwhelming reviewers.”

— Nikolov et al., Google, January 2025

An organization with a review-shaped oracle throttles generation to protect the oracle. That is not a failure of the tooling. It is the only rational response to the architecture.

36.3 Uber: The Machine Decides, at Population Scale #

Uber’s Spark upgrade covers a different unit of work. Roughly 2 million Spark applications launch daily through more than 20,000 scheduled workflows, drawing on more than 2,100 applications of Spark source code.¹⁶⁸ The three numbers are stated separately in the source and never reconciled; the 2 million is a daily execution count, not two million distinct programs, and the distinction matters when comparing against Google’s change counts.

Code transformation used Polyglot Piranha, an open-source abstract-syntax-tree rewriter Uber originally built for stale feature-flag cleanup, extended with structural rules for the Spark 2 to Spark 3 transition. No language model appears anywhere in the account. That is worth stating precisely: the post does not argue against using one, it simply never raises the possibility.

The oracle is the interesting part, and it is nothing like Google’s. Uber built a framework called Iron Dome that validates a transformed job by running it against production data and comparing what it wrote. Spark’s catalog interface and Hadoop’s file output committer are intercepted so that a production path /db/tbl is rewritten at runtime to /stgdb/tbl. Guardrails at the filesystem interface prevent accidental production writes. The interceptors publish telemetry naming every table and path the job touched, and that telemetry is used to validate output against the production counterpart. Orchestration then shadows production runs, validates the data, and marks jobs for migration automatically.¹⁶⁸

No human reads a diff. The oracle is output equivalence on real data, checked by machine, for every artifact.

Reported outcomes: an “85% job migration” heading, with the sentence beneath it stating that the majority of the 20,000 jobs transitioned within six months; more than 60 percent of jobs improving by more than 10 percent in performance; and eventual completion at 100 percent of Spark applications upgraded to 3.3, with a 50 percent reduction in runtime and resource usage overall.¹⁶⁸ The fraction of the 2,100 source applications that Piranha transformed without manual work is not published, and should not be inferred from the 85 percent, which is a completion figure rather than an automation rate.

36.4 The Same Migration, Both Ways #

The Spark comparison is imperfect because the units differ. A second Uber account closes that gap, because it covers the same migration family Google reported—JUnit—and states outright that language models were tried and set aside.

Uber migrated more than 75,000 test classes from JUnit 4 to JUnit 5, modifying over 1.25 million lines within four months, against a denominator of more than 600,000 JUnit 4 tests across 15 million lines. The transformation ran on OpenRewrite recipes operating over a lossless semantic tree, integrated with the build system through custom Bazel aspects, producing over 5,000 diffs.²³⁴

The key learning is stated in three sentences:

“We attempted to use generative AI to migrate multiple test class files at once, but this approach was unsuccessful. AI usage was limited to debugging test and build failures. Given our codebase’s scale and custom testing patterns, a deterministic approach proved more practical than an AI-driven migration.”

— Uber Engineering, April 7, 2026

Note that this is precisely the inverse of Google’s split. Google puts the model on edit generation and the deterministic machinery on location and validation. Uber puts the deterministic machinery on edit generation and the model on failure triage.

The failure-handling policy follows from the scale: “For test failures, we reverted affected files back to JUnit 4. Given the scale spanning thousands of targets, diagnosing individual test failures was impractical.” Recurring build failures were addressed by writing additional recipes—fixing the rule rather than the instance.²³⁴

For comparison, Google’s JUnit 3 to JUnit 4 migration covered 5,359 files and more than 149,000 lines in three months, with approximately 87 percent of model-generated code committed unchanged.¹⁰⁷

36.5 What the Pair Shows #

The generative capability is not what separates these programs. What separates them is that one has an oracle a machine can run on every artifact and the other has an oracle made of people.

Where a cheap mechanical decision procedure exists—output equivalence, differential comparison against a reference, a rule that is correct by construction—determinism scales to population size and the model is best used on the residue. Where no such procedure exists, the model can still generate at volume, but throughput is capped by review, and the responsible organizations throttle generation to match. Google did exactly that and said so.

This is the same finding as Section 6.1, arrived at from the opposite direction. Delegate by verifiability, not by difficulty.

36.6 Two Cautions #

The unit gap is partly an artifact. Google counts changelists and edits authored; Uber counts jobs executed and validated. A Spark job cleared by an output diff after a configuration-injection rule is not the same quantum of work as a semantic type migration through a call graph five hops deep. The honest contrast is not 595 against 20,000. It is what sits in the validation loop.

Uber is not the deterministic foil this pairing might suggest. By August 2026 the same organization attributes more than 70 percent of its pull requests to local or cloud agents, runs more than 3,600 agent skills executing over 30,000 times a day, and reports weekly agent requests up 9.4 times in six months.²⁴² Both accounts above predate or coincide with that build-out, and the JUnit post keeps the model in the loop for failure triage. The defensible reading is method selection by problem class, not a verdict on agents.

36.7 What Transfers #

Before delegating a migration to agents, answer one question: does a mechanical check exist that can decide correctness for every artifact without a person reading it? Shadow execution with output comparison, differential testing against the prior version, and a transformation that is correct by construction all qualify. Build-plus-tests qualifies only where the suite is strong enough that passing means something.

If such a check exists, build it first and let it govern the rollout, whatever generates the edit. Uber’s Iron Dome is the reusable artifact from that migration, not the Piranha rules; the post says so, noting the framework “paved the way for future Spark upgrades.”¹⁶⁸

If no such check exists, size the program against reviewer capacity from the first week, and throttle generation deliberately rather than discovering the ceiling by overwhelming a team. Google’s throttling is the practice to copy.

36.8 What This Rests On #

The Google figures are peer-reviewed and carry the authors’ own generalizability warnings: a single organization’s infrastructure, a single migration type, a model fine-tuned on internal code, and effort measured by edit distance as a proxy. The 50 percent time saving is developer estimation, not instrumentation. The Uber figures are first-party and unaudited, and the Spark post’s headline percentage is a section heading rather than a measured automation rate. Neither organization publishes a defect-escape rate, so no claim about relative correctness is available from either side—only about relative throughput and about what decided correctness along the way.