Budgeting and Instrumentation

25.1 Cost Is Stochastic, Not Linear #

One finding breaks naive capacity planning outright: runs on the same task can differ by up to thirtyfold in total tokens.⁶

The same work established two further findings that destroy the intuitive planning approach. Accuracy often peaks at intermediate cost and saturates above it, so spending more does not monotonically buy correctness. And two separate results say that cost is hard to anticipate. Models forecast their own token consumption poorly, with correlations up to 0.39 and a systematic bias toward underestimating. And expert-rated task difficulty—SWE-bench-Verified’s estimate of how long a professional developer would need to resolve the issue—is only a weak predictor of what an agent actually spends.⁶ Keep those distinct: the second measures whether human effort categories predict agent tokens, and no one in that study was asked to forecast an agent’s bill. It is a reason to distrust difficulty as a proxy, not evidence about how well an expert could forecast agent cost if asked, which was not tested. Weak is also not useless—the categories carry some signal, and coarse signal beats none when you are sizing a budget.

The consequence is that per-task budgeting must be distributional rather than a point estimate. A mean is not a plan. Track p50, p95, and p99, and set alarms on the tail.

Spending more also does not guarantee scoring better. Two entries recorded from the same public leaderboard make the point: one model-scaffold combination at $366.81 for a 27.2 percent resolve rate against another at $67.09 for 38.0 percent.⁵⁵ Two points refute “the dearer system wins”; they say nothing about the correlation across the population, which two observations cannot estimate at all. The leaderboard route they were read from no longer resolves either, so treat the pair as an illustration and the live board as the thing to re-read. Paying more is not a strategy.

25.2 The Minimum Instrumentation #

Four things matter here that most organizations ignore completely.

Those four become one record written at the end of every session. The type below carries them as termination, cost_usd, turns, and a computed cache_hit_rate, alongside four correlation keys that make the population queryable: the task class, so distributions can be compared like with like; the model snapshot, so a vendor-side change is separable from your own (§33.5); the prompt fingerprint, so a quality shift can be tied to a prompt edit (§13.3–13.4); and wall time, so latency regressions surface without a second pipeline.

from dataclasses import dataclass, field
from enum import Enum

class Termination(str, Enum):
    COMPLETED       = "completed"
    MAX_TURNS       = "max_turns_exceeded"
    BUDGET          = "budget_exceeded"
    TOOL_ERROR      = "execution_error"
    VALIDATION      = "validation_exhausted"
    ABORTED         = "aborted_on_divergence"

@dataclass
class SessionRecord:
    session_id:   str
    task_class:   str
    provider:     str          # which usage shape `usage` carries (§22.8)
    model:        str
    prompt_fingerprint: str
    termination:  Termination
    turns:        int
    usage:        dict = field(default_factory=dict)
    cost_usd:     float = 0.0
    wall_ms:      int   = 0

    @property
    def cache_hit_rate(self) -> float:
        # `usage` holds the provider's native fields; §22.8's adapter is what
        # makes reads/writes/fresh disjoint, and it runs here. Never read the
        # raw fields directly, since OpenAI's `input_tokens` already contains
        # the cached ones and summing them double-counts.
        reads, writes, fresh = normalize(self.usage, self.provider)
        total = reads + writes + fresh
        return reads / total if total else 0.0

None of those four fields is useful on a single session. They are population statistics, and that is the point: you are not debugging one run, you are characterizing a distribution. A single expensive session tells you nothing because §25.1 established that cost varies up to thirtyfold on identical work. The same session sitting at the ninety-ninth percentile of its task class tells you a great deal.

If you remember nothing else from this chapter, remember the typed termination reason. It costs one enum, it is the field organizations most reliably omit, and it converts “the agents seem worse this week” from an impression into a query.

25.3 Which Alarms to Set #

The third one is provider-specific and frequently missed. On a model with a context threshold, crossing it raises the rate on every request that stays above the line. Alarming on the crossing rather than on the resulting bill gives you a chance to intervene.

25.4 Runaway Detection #

Mechanisms exist, and the evidence for these mechanisms does not. Turn caps, budget caps, concurrency and spawn-depth limits, execution-time ceilings—all real, all implementable, none with a published measurement of how often production agents actually enter non-terminating loops, no loop-specific detector with a published false-positive rate, and no published cost-anomaly threshold.

Be precise about what has been measured because something adjacent has. The trajectory work behind §4.3 reports a general failure monitor at roughly 2 to 3 percent false positives and about 82 percent precision at flagging a run that has already locked in, with recall under thirty percent and a median lead time of zero.¹⁰ Note which event that lead time is measured against: t_lock, the point after which the paper observes no successful recovery, and not the decisive error, which precedes lock-in. Only 3.7 to 8.7 percent of failures are flagged before lock-in at all. That is a confirmation instrument rather than a runaway alarm—it tells you a run has failed, not that one is about to—and it is the nearest thing to a calibrated number in this area.

Build it, instrument it, and do not claim a benchmark you do not have. [unmeasured]

class RunawayGuard:
    """Both dimensions. At least one major SDK defaults both to unlimited."""
    def __init__(self, max_turns=40, max_cost_usd=5.00, max_wall_s=1800):
        self.max_turns, self.max_cost, self.max_wall = max_turns, max_cost_usd, max_wall_s

    def check(self, turns, cost_usd, elapsed_s) -> Termination | None:
        if turns    >= self.max_turns: return Termination.MAX_TURNS
        if cost_usd >= self.max_cost:  return Termination.BUDGET
        if elapsed_s >= self.max_wall: return Termination.BUDGET
        return None

The early-abort case has better support. Eighty-two percent of failed trajectories keep executing after the point at which the failure is already unrecoverable—repairing the wrong problem, retrying a strategy that cannot work, or verifying an outcome that can no longer change—and the first observable signal surfaces roughly ten steps after the decisive error, which is the earlier of the two events.¹⁰ An agent that is going to fail is cheapest to stop early, and a long trajectory is a negative signal.

25.5 Why Uniform Caps Are the Wrong Response #

The most common reaction to a cost surprise is a uniform per-developer cap. It is reactive by definition and it cuts the wrong spending first.

The reason is that the headroom is widest precisely where acceptance is worst. If a team’s first-pass acceptance rate is 30 percent, cost per accepted outcome is 3.3× the cost of an attempt; at 85 percent it is 1.18×. Closing that gap is the ceiling on what better context, a stronger model or more verification could return, and a uniform cap constrains both teams identically while leaving the larger ceiling untouched.

Headroom is not the same as marginal return, and conflating the two gets allocation backward. It says nothing about what the next dollar buys, because raising acceptance usually raises attempt cost too, and the ratio is what matters:

The wide-headroom team got materially worse and the narrow-headroom one got better. So treat the multiple as potential worth investigating where acceptance is poor, and allocate on the measured change in cost per accepted outcome rather than on the headroom figure.

The alternative is spend shaped by task class and measured by outcome rather than by seat:

BUDGETS = {
    "lint-fix":        {"model": "claude-haiku-4-5", "max_cost": 0.10, "max_turns": 8},
    "test-repair":     {"model": "claude-sonnet-5",  "max_cost": 0.75, "max_turns": 20},
    "feature":         {"model": "claude-sonnet-5",  "max_cost": 3.00, "max_turns": 40},
    "architecture":    {"model": "claude-opus-5",    "max_cost": 8.00, "max_turns": 60},
    "incident-debug":  {"model": "claude-opus-5",    "max_cost": 15.00, "max_turns": 80},
}

That is a policy someone can defend in a budget conversation because each line is a claim about the value of the work rather than an arbitrary ceiling on a person. Construct §25.4’s guard from these per task class rather than relying on its defaults, which are tighter than the two largest classes here allow.

25.6 What a Sound Budget Looks Like #

Everything in this chapter converges on a single artifact: a spend policy someone can defend in a budget review without reaching for a percentage they cannot source.

A budget that holds up has four properties, and each comes from a preceding subsection. It is distributional, expressed as p50 and p95 by task class rather than as a mean because cost varies up to thirtyfold on identical work (§25.1). It is bounded on two dimensions, turns and dollars because at least one major SDK leaves both unlimited by default (§25.4). It is shaped by task value rather than by seat because a uniform cap constrains a team with wide headroom and one with none identically, and because where additional spend actually returns is a measured question rather than one headroom answers (§25.5). And it is denominated in accepted outcomes, not attempted ones, because a 40 percent cut in per-attempt cost that drops first-pass acceptance from 85 to 45 percent raises the real cost by 13 percent while lowering the invoice (§21.5). Denominate that cut in money rather than tokens: the lanes price differently, so a 40 percent cut in token count that falls mostly on output can cut the bill by two-thirds and come out ahead even on those acceptance numbers.

The last of these is the property that changes conversations. A team asked to cut agent spend by a third can always do it. Whether they should is a question the invoice cannot answer and cost-per-accepted-outcome can.

One lever is large enough to warrant its own chapter. Model tiers differ by roughly twenty to one on output price, so routing mechanical work to a small model is a five- to tenfold saving on that traffic at no quality cost, and unlike every other control here, it requires no one to change how they work. §27 covers the strategies, the published savings and what they were actually measured on, why agentic workloads are the hard case, and how to instrument it.