Tooling Evaluation Criteria

This framework is deliberately vendor-agnostic, and this section is where that posture becomes actionable.

Two distinct question sets are needed and they are usually collapsed into one. The first is universal: a set of governance properties that any AI participant in delivery must satisfy regardless of what stage it serves, because they determine whether the organization can attribute, bound, revoke, and audit what the tool does. The second is stage-specific: the properties that actually differentiate two tools serving the same lifecycle stage, and which look entirely different for a requirements platform than for a pipeline agent. A procurement process that asks only the universal questions buys governable tools that do not fit the work. One that asks only the stage-specific questions buys capable tools that cannot be governed. Ask both.

21.1 Capability Classes by Stage #

Distinguish these classes, because they carry different risk, bind at different stages, and are frequently bundled and sold as a single product.

Two observations from this table are worth carrying into a buying decision. Most enterprise attention and most vendor investment sit in the generation-heavy classes (inline assistants, repository-resident agents, and pipeline agents), while the evidence in Section 3 places the binding organizational constraint in the two classes that absorb their output, review and analysis and test and evaluation. And the first four classes, which govern what everything downstream is actually working from, are the least likely to be evaluated as AI tooling at all, because they were usually bought years ago for other reasons.

21.2 Universal Criteria #

These apply to every class in the table above and are scored pass or fail, not weighted. A tool that fails several is ungovernable at any stage, however well it performs. Three carry evidence from elsewhere in this framework: automatic attribution that a user cannot correct corrupts the record it creates, which is one reason at least one major editor ships its equivalent setting defaulting to off;¹⁶ licensing exposure from training-data reproduction is small but non-zero and scales with context size, which is the direction agentic tooling is moving;^49,50 and the measured cross-product merge conflict rate is roughly double the same-product rate, so multi-vendor strategies carry a cost no vendor comparison will surface.³²

21.3 Stage-Specific Criteria #

These are weighted rather than pass-fail, because their relative importance depends on where the organization’s constraint sits. Score each 2 (met and demonstrated), 1 (partially met or met by roadmap), or 0 (not met). An unanswered criterion scores 0; the absence of an answer is data.

Three criteria below rest on evidence stated earlier: between 36.7 and 55.7 percent of tasks in one study carried a known CVE in the dependency version the model specified, which is why S4-2 exists;⁶⁴ the capabilities that most determine whether an organization realizes value are organizational rather than product properties, which is why S1-4 is a caution rather than a criterion;⁹⁸ and published guidance exists on the security content that belongs in agent instruction files, which is what S3-5 scores against.⁹⁹

21.4 Scoring and Use #

• Run U-1 through U-11 first, as gates. They are pass or fail. A vendor failing three or more should not proceed to stage scoring, because the stage capability will not survive the governance gap.
• Score the stage criteria only for the stages the tool actually serves, using the capability-class table in 16.1 to decide which apply. Scoring a requirements platform against S6 produces a low number that means nothing.
• Weight stage fit by where your constraint actually is. Most organizations shop for stage four generation speed. The evidence in Section 3 places the binding constraint in stage five verification capacity and in the review load that stage four generates. Weights that mirror the evidence rather than the sales motion produce a different shortlist, which is the point of scoring at all.
• Weight the universal criteria above capability. Capability improves rapidly and converges across products; governance architecture is set at design time and rarely changes. A tool that generates excellent output and cannot be governed will be replaced within eighteen months at considerable cost. A tool that generates adequate output inside a sound authorization model will still be serviceable.
• Resolve the suite-versus-best-of-breed question deliberately. The concurrency finding argues for fewer agent products operating on the same repository. The portability and exit criteria argue against a single vendor owning the measurement layer, because a vendor that both generates and grades is not an independent verifier. The reconcilable position is to consolidate the things that write and diversify the things that measure.
• Demand demonstrations for U-1 through U-5 and for S4-1. All six are testable in an afternoon, and all six are areas where vendor documentation and shipped behavior have diverged in public. Score a documented-but-undemonstrated claim as 1, never 2.
• Record vague answers as findings rather than as zeros. A zero says the capability is absent; a finding says the vendor could not describe its own product’s behavior, which predicts the support relationship. The pattern worth watching for is a confident answer about the platform’s general security posture offered in response to a specific question about agent behavior.