Stage Seven — Operations and Maintenance

16.1 What Changes at This Stage #

Operations inherits every deferred decision from the preceding six stages, and it inherits them at a scale that was set elsewhere. Three shifts matter most.

Validation stops being a gate and becomes a process. This is the one place where a published international standard has already made the argument. ISO/IEC 5338 classifies AI system life cycle processes as generic, modified, or AI-specific relative to the underlying software and systems lifecycle standards, and identifies exactly three AI-specific processes: knowledge acquisition, AI data engineering, and continuous validation.⁶ Continuous validation formalizes the position that validation is not something a system passes once but something that runs for its operational life, because behavior drifts. For an agentic SDLC framework this is the standards-body precedent for the claim that the release boundary is no longer where verification ends. Cite it; it is more durable than any argument this document could construct.

Four distinct phenomena get collapsed into the word “drift,” and they need different monitoring. Model drift is provider-side: the model changes beneath a stable API surface. The canonical demonstration measured one model’s accuracy on a specific task falling from 84 percent to 51 percent across three months, alongside reduced instruction-following and increased code-formatting errors.⁸⁷ That paper drew methodological criticism at the time, and the criticism is fair on the specific task, but the core claim—that a production model service’s behavior can shift materially within months without a version change visible to the caller—has not been refuted. Data drift is input-side and is the classical machine learning problem, with the classical treatment still applying.⁸⁸ Prompt drift is accumulated uncoordinated edits to prompts and system instructions, often across teams, usually with no test coverage; it is real, widely observed, and has essentially no research literature, so the mitigation—version prompts as code, require an evaluation diff on change, treat a prompt edit as a deployment—is sound engineering reasoning rather than an evidenced practice, and should be presented internally as such. Evaluation set staleness is covered in Section 14.

Correctness becomes unobservable at request time. Classical service level objectives assume a binary per-request outcome. For an agent, you frequently cannot tell whether a response was correct without a downstream signal or a human. That breaks the error budget model at its foundation, because a budget cannot be debited for failures that are not detected. The workable partial answer is to define indicators on observable proxies and to run correctness as a sampled offline evaluation feeding a separate quality metric. The best-specified public proposal defines seven indicator types—task success rate, tool call accuracy, 95th-percentile latency, cost per task, policy compliance, scope chain depth, and hallucination rate—with conventional burn-rate alerting, a cost-guard kill switch triggering at 95 percent of budget consumption, and a per-agent circuit breaker.⁸⁹ Two of those indicators are genuinely novel and worth adopting whatever the implementation: scope chain depth, measuring how far an agent has traversed from its authorized boundary, and cost per task treated as a reliability signal rather than a finance metric. That work is published open source in public preview, with no production efficacy data, and should be treated as a well-specified proposal rather than established practice.

16.2 Elements to Adopt #

Telemetry that covers the agentic layer. The transport is settled and the semantics are not, and being precise about that distinction saves an architecture argument. The OpenTelemetry generative AI semantic conventions have moved to a dedicated repository, and every span, event, metric, and gen_ai.* attribute is marked Development status with no tagged release—the only stable attributes appearing in these signals are inherited core ones.⁹⁰ Anyone claiming the conventions are stable is wrong. They are nonetheless the right thing to emit, with attribute names treated as a versioned contract with your own pipeline. Two design decisions are worth knowing. The conventions define a distinct planning operation alongside agent invocation and tool execution, which makes an agent’s task decomposition separately observable from what it then does. And an evaluation result event carries an evaluation name and a score value, which is the first credible standard bridge between the evaluation layer and the observability layer. Message content attributes are opt-in by default for privacy reasons and should be enabled under a data classification policy rather than globally.

Per-agent, per-task cost observability. Cost is a reliability and security signal, not only a finance one. Runaway consumption is the earliest observable indicator of an agent in a loop, an agent being manipulated, or an agent operating outside its intended scope. The unit metric is cost per task, not cost per token, precisely because agentic workloads consume far more tokens per unit of work than conversational use and unit-price declines therefore do not imply bill declines. The relevant billing standard currently handles AI through existing columns—consumed quantity and unit carrying token counts, SKU identifiers marking token charges—rather than through AI-native fields, which is workable and worth knowing is a stopgap.⁹¹

Kill switches that are tested and measured. For every agent operating against production, a kill switch that halts the individual agent, the agent class, and the whole agent estate, each with a documented and measured time-to-effect. Untested kill switches are a documentation artifact. The specific gap to check for is agents running inside SaaS platforms, third-party tools, and developer environments, where the orchestrator’s kill switch does not reach.

Incident response that accounts for unreliable self-report. Add three items to the runbook. First, a discriminator between model failure and code failure: re-run the same input against the pinned model and the deployed code, where reproducible failure implicates code or prompt, non-reproducible failure implicates model non-determinism or provider-side change, and failure that reproduces against a new snapshot but not the old one implicates a provider change. This is sound reasoning rather than published practice and should be labeled as such. Second, evidence preservation from ephemeral agent runtimes, including snapshot-on-anomaly before a container is destroyed, because the ephemerality adopted as a security control destroys the forensic record. Third, explicit guidance that an agent’s account of its own actions is evidence to be verified rather than testimony to be believed.

Maintenance practice that addresses comprehension. The maintainability data in Section 3.6 and the comprehension finding in Section 2 point at the same operational risk from different directions: systems accumulating code that nobody fully understands, with debt persisting at measured rates and security issues persisting nearly twice as long as other categories.³⁹ Practical responses: require an explanatory record for non-trivial agent-authored change at the time of authorship rather than at the time of confusion; schedule and fund consolidation work explicitly, since the data says it will not happen spontaneously; measure duplication and reuse as operational metrics rather than aesthetic ones; and maintain named human ownership of every subsystem with a stated expectation that the owner can explain it.

Non-human identity lifecycle discipline. Agent identities are provisioned quickly and deprovisioned rarely. Bring them into access review at the same cadence as privileged human accounts or more frequently, given their number and rate of change, and report non-human and human identity governance separately. A combined figure conceals a much worse machine posture behind good human hygiene, and the improper-offboarding failure that leads the published non-human identity risk list is exactly what a combined figure hides.⁶⁹

Continuous validation in production. Run the evaluation suite against production traffic on a schedule, not only at release. Alert on distribution shift rather than on errors. This is the operational implementation of the ISO/IEC 5338 process and the only reliable detector of provider-side model drift.

16.3 Controls #

16.4 Evidence to Request #

• Kill switch test results with measured time-to-effect, by agent class, including at least one agent operating inside a third-party platform.
• Telemetry coverage: what fraction of the agent estate emits action telemetry, and what fraction of what is emitted is actually analyzed.
• The most recent continuous validation run against production, with its distribution comparison.
• A cost anomaly and the response it triggered.
• The non-human identity access review, reported separately from human identities.
• Duplication and reuse trend over the last four quarters, with any consolidation work that was actually funded.

16.5 Failure Modes #

• Recording mistaken for detection. Comprehensive agent telemetry that nobody and nothing analyzes is an evidentiary asset, not a control. Instrument the generated-to-analyzed ratio explicitly, because it degrades silently.
• Cost visible only in aggregate. A monthly bill with no per-agent or per-task attribution cannot detect a looping agent, cannot support a chargeback conversation, and cannot inform a tier decision.
• Drift monitored only at version boundaries. The organization watches for announced model retirements and does not watch for behavioral change within a pinned snapshot, which is the drift that actually surprises people.
• Kill switches that do not reach. They exist in the orchestrator and not for the agent embedded in a SaaS platform, the agent in a partner’s environment, or the agent running on a developer’s laptop against corporate credentials.
• Comprehension debt as a permanent unbudgeted liability. It produces no ticket and no alert. It surfaces as an inability to change something, usually urgently, usually during an incident, and by then the people who could have explained it have moved on.