Fleet Maintenance at Scale

38.1 Context #

Spotify runs a polyrepo layout: thousands of GitHub repositories holding small, independently deployable components owned by individual squads. Roughly 60 million lines sit in production components. The condition that forced the investment is stated plainly—the production codebase was growing seven times faster than the engineering headcount.¹⁰¹

38.2 The Mechanism #

The unit of change is not a patch. It is a container image.

A “shift” is a Kubernetes custom resource stored in GitHub that names a Docker image containing the transformation logic, the target repository set, and the pull request metadata. The platform clones each repository, executes the image against the checked-out code, records the file changes, creates a commit, and opens a pull request. Targeting is done by GitHub search or by querying ingested source code in a data warehouse. Because the transformation is a container, it can be anything—stream editors, purpose-built code, OpenRewrite, Scalafix—and the platform does not care which.²³⁵

Merge authority is inverted. An automerger service merges pull requests that pass all tests, which moves the decision from the repository-owning team to the change author. Two guards constrain it: automerge runs only during the owning team’s working hours and never on weekends or holidays, and risky changes roll out in cohorts, with each cohort gated on the previous one succeeding. A service consuming deployment and pipeline-execution events correlates failures with recently automerged changes and feeds that signal back into the gating.²³⁵

None of this used a language model. The 2023 account describes ordinary tooling and custom code, and mentions no AI at all.

38.3 The Oracle #

The repository’s own tests, plus production telemetry as a second-order check.

That is thinner than the Uber Spark oracle in Section 36 and much thinner than Meta’s in Section 37, and the compensating mechanism is temporal rather than logical: cohort gating means a defective change is caught after it has damaged a small number of repositories rather than before it has damaged any. The working-hours constraint exists for the same reason—so that a human is present when the cohort fails.

This is a legitimate design. It is not the same thing as verification, and an organization adopting it should be clear about which one it is buying.

38.4 What Was Measured #

In 2022, the fleet platform created more than 270,000 pull requests, of which 77 percent were automerged and 11 percent merged by a human. The 241,000 merged changes represented 4.2 million lines changed.²³⁵ Time to reach 70 percent fleet adoption of a backend framework release fell from about 200 days to under seven. During Log4j, 80 percent of backend services were patched within nine hours.

By June 2026 the cumulative figure was “more than 2.5 million automated maintenance PRs,” described as “the vast majority auto-merged with no human in the loop.”¹⁰¹ No start date is given for that total, so it should be cited as cumulative-to-2026 rather than attributed to any single period, and the boundary between deterministic and agent-generated changes within it is not published.

The agent layer arrived later, with an explicitly stated purpose: to lower the barrier to entry so that fleet management could handle changes too complex to express as a rule.¹⁰⁰ By November 2025 it had produced more than 1,500 pull requests merged into production, with reported time savings of 60 to 90 percent against writing the same code by hand and hundreds of developers interacting with it. No merge rate—merged over opened—is published for the agent at any point in the four-part series, which is a conspicuous omission given how carefully the deterministic platform’s rate was reported.

One worked deployment is reported in full: roughly 1,800 downstream data pipelines affected by deprecated datasets, 240 automated migration pull requests deployed over six months across three pipeline frameworks, against an estimate of about ten engineering weeks to do the same work manually.²³⁸

38.5 What Failed #

The engineering team’s own account of what did not work is the most useful part of the series.

Their homegrown agent loop, capped at ten turns per session with three session retries, exhausted its turns on multi-file cascading changes. Open-source agent frameworks “struggled producing reliable, mergeable PRs at scale.” Requiring users to name exact files created friction. Tool proliferation introduced unpredictability, and limiting the agent’s tool access improved reliability—the agent was eventually given a verification tool, a restricted git tool, and a shell tool on a strict allowlist, with code search and documentation tools deliberately withheld.²³⁶

A model judge vetoes roughly 25 percent of agent sessions, and about half of vetoed sessions result in self-correction. The stated reason is worth quoting because it names a failure mode most organizations do not instrument: “some agents were a bit too ‘ambitious’, trying to solve problems that weren’t strictly in their prompt.”²³⁷

Their ranking of failure modes matches this framework’s: an agent that produces no pull request is an annoyance; one that produces a failing pull request is frustrating; one that produces a pull request that passes continuous integration and is functionally wrong is the serious case, because it erodes trust.²³⁷

Verification tooling supports Linux on x86 only, which excludes iOS and ARM work entirely.

And the aggregate effect on the organization is stated without spin: a 76 percent increase in pull request frequency means “76% more PRs to review.”¹⁰¹ Generation scaled. The review function absorbed all of it.

38.6 What Transfers #

The structural finding is the sequencing. Seven years of deterministic infrastructure—targeting, cohorting, automerge policy, failure correlation—is what made an agent deployable here. The agent inherited a rollout and verification system it did not have to build. An organization without that substrate that adopts a background agent is not reproducing this case; it is reproducing the first year of it.

The tool-restriction finding transfers directly and is cheap to apply: fewer tools produced more predictable agents, and the tools withheld were the open-ended search ones.

The judge-veto rate is worth adopting as an instrument even where the judge itself is weak. Twenty-five percent of sessions attempting something outside their brief is a scope-discipline measurement, and almost nobody computes it.

The automerge design transfers only with its guards intact. Automerge without cohort gating, without working-hours constraint, and without failure correlation is not this system.

38.7 What This Rests On #

Everything here is first-party and self-reported. The 2.5 million figure has no published time window and no published composition. No failure rate, rollback rate, or defect-escape rate is published for automerged fleet changes at any point in the series, which means the safety of the automerge design is asserted rather than demonstrated. The agent’s merge rate is unpublished. The 60 to 90 percent time savings is an estimate.