Agent Testing

Why Agent Testing Is Different

Classical software testing assumes deterministic output — same input, same result. AI agents are stochastic, multi-step, and tool-using. A unit test that asserts assertEqual(result, expected) cannot evaluate whether the agent took the right reasoning path, used the right tool in the right order, or made a decision that will hold at scale. Three dimensions must be tested independently: output quality, reasoning trajectory, and operational safety.

The 10.6% → 52.1% lesson: an agent can return a plausible-looking answer that is semantically wrong. Output scoring alone misses this. The Databricks Genie 9-Rung Ladder shows that 42% of accuracy gains came not from a better model, but from adding governed context. Testing must verify the context is being used correctly, not just that a number appeared.

Scoring Dimensions

Generation 1

Output Scoring

Did the agent return the right answer? Compares final response to a reference. Fast and cheap. Misses correct answers via wrong paths and wrong answers via lucky data alignment. The Rung 8 benchmark suite operates here.

Generation 2

Trajectory Scoring

Did it use the right tools in the right order with the right arguments? A correct answer via a wrong path (e.g. paid_lr matching ultimate_lr by coincidence) is flagged as a brittle pass. LangSmith, W&B Weave, TruLens, and Vertex AI all treat this as first-class.

Generation 2

Ontology-Derived Evals

SHACL constraint → auto-generated test case. If a constraint says earnedPremium > 0, a trap question is generated automatically: "what if earned premium is zero?" The ontology IS the test spec. Tredence's agentic repair loop uses this pattern.

Emerging

Multi-Agent Eval

No deployed standard yet. Compositional failure: each sub-agent passes individually; the orchestration fails. arXiv 2609.18820 ("Compositional Policy Violations") shows step-level compliance can pass while workflow-level compliance fails.

LLM-as-Judge vs Agent-as-Judge

Using a language model to score another language model's output is Generation 2's key method — but its reliability is contested.

Reliability risk

LLM-as-Judge

AgentRewardBench (COLM 2025): 1,300+ expert-annotated web-agent trajectories. Finding: no single LLM judge reliably scores across tasks. Preference leakage (Li et al. 2025) contaminates scoring when the judge's own training overlaps with benchmark data. ~70% agreement with human experts.

Better

Agent-as-Judge

Zhuge et al. (2025): a separate agent evaluates the primary agent's reasoning steps, not just its final answer. Achieves ~90% agreement with human experts vs. ~70% for simple LLM-as-judge. The evaluator agent itself can be tested and versioned separately from the agent under test.

New (2026)

TRAIL Framework

Trace Reasoning and Agentic Issue Localization (arXiv 2505.08638). Goes beyond pass/fail: diagnoses exactly where in a trace the agent deviated. "The agent had the right tool but called it with the wrong policy date" — issue localized to step 3 of 7.

Known gap

Benchmark Contamination

Search-time contamination: web-enabled agents retrieve benchmark answers during execution. SWE-bench saturation: OpenAI (Feb 2026) argues the benchmark no longer measures frontier coding capability. Preference leakage from overlapping judge training data. Static benchmarks have a shelf life.

Eval-Driven Development (EDD)

The agent testing equivalent of Test-Driven Development. Write the eval before writing the fix. Because agents are non-deterministic, assertions need statistical thresholds, not binary pass/fail.

Rule

What it means

Why it matters

Write eval first

Define the benchmark question and success threshold before changing the agent or ontology

Forces you to state what "correct" means before being influenced by the output

Statistical gates

Pass if score ≥ X% over N runs, not assertEqual on a single run

Stochastic agents have variance; a single run result is not a proof

Token budget

Token usage must stay within 5% of baseline (FutureAGI guidance)

A fix that doubles token cost is not a free fix — latency and cost regress

Monitor in prod

Run evals in CI and in production monitoring — not only pre-merge

Pass rate drifts when nothing in your repo changed (provider silently updates model)

GitOps

Ontology PR triggers full eval suite before merge

A YAML metric change that breaks a downstream agent is caught before it ships

Testing Maturity Model

Level 1

Output check

"Did the agent return 52.1%?" — Rung 8 benchmark suite

Level 2

Trace check

"Did it use ultimate_lr, not paid_lr?" — MLflow Traces, LangSmith, W&B Weave

Level 3

Ontology-derived

SHACL rule → auto-generate test case — Tredence agentic repair loop

Level 4

Adversarial eval

LLM/agent generates trap questions from the schema — Braintrust, RAGAS, custom harnesses

Level 5

Continuous regression

Ontology PR → eval suite in CI/CD — GitOps for ontology (emerging 2026–27)

Core principle: The benchmark suite is not a certificate of quality — it is a regression harness. Its job is not to prove the agent is good; it is to catch when the agent gets worse after the next change to the ontology, the model, or the data.

Breaking — OpenAI hosted Evals shutting down: OpenAI's Agent Builder and hosted Evals platform goes read-only 31 Oct 2026 and is removed 30 Nov 2026. Official migration recommendation: Promptfoo (OpenAI cookbook, Jun 2026). The open-source openai/evals repo continues as a community benchmark registry but is not a trajectory evaluator and ships no CI/CD runner.

Framework Comparison

Framework	Tests	How it works	Licence / cost	Best for
DeepEval	Output, RAG, agent task, span-level	pytest-style runner, 50+ metrics (G-Eval, faithfulness, task completion). CI-friendly. Confident AI is the hosted wrapper.	Apache 2.0 (open source)	Most popular OSS option; teams wanting CI/CD integration out of the box
LangSmith	Output, trace, trajectory, tool calls, multi-turn	Hosted; integrates with LangGraph for span-level and full-trace scoring; trajectory evals are first-class	Proprietary ~$39/seat/month	LangChain / LangGraph shops; teams that need trajectory scoring today
Braintrust	Output, span, full trace, A/B prompt	Hosted; eval + observability in one; MIT-licensed proxy; scorers run per span or completed trace; versioned prompt/eval management	MIT proxy / hosted pricing	Teams wanting versioned eval management and A/B prompt testing
W&B Weave	Agent traces, conversations, LLM calls, tool calls	Apache 2.0 toolkit; ingests OpenTelemetry spans; multi-modal trace support	Apache 2.0 (open source)	Teams already using Weights & Biases for ML experiment tracking
TruLens	Agent traces, logical consistency, tool calling, goal-plan-action alignment	Open source; reference-free goal-plan-action alignment metrics; CI/CD gates for Azure Pipelines, CircleCI, GitHub Actions. MLflow integrates TruLens scorers natively (2026).	Apache 2.0	Azure / MLflow shops; multi-step reasoning validation
RAGAS	RAG pipeline only (retrieval, faithfulness, relevancy)	Open source; specialized — not built for general agent or multi-turn eval	Apache 2.0	RAG pipeline quality only; combine with DeepEval for agent evals
Promptfoo	Security, prompt injection, red-teaming, vulnerability scanning	Open source CLI + CI integration; adversarial probing; now OpenAI's recommended migration target	MIT (open source)	Security testing, red-teaming, prompt injection harness
Inspect (UK AISI)	Safety, autonomy, cyberattack capability, tool use, multi-turn	MIT-licensed Python harness; METR, Apollo Research, and government testing bodies run on it. Inspect Hawk = METR's AWS infrastructure layer.	MIT (open source)	Safety evals, government / regulated-sector compliance, frontier risk assessment
Langfuse	Traces, spans, dataset evals	Open source + self-hosted option; privacy-first; no data leaves infrastructure	MIT (open source)	Privacy-sensitive deployments where traces cannot leave your infrastructure
Phoenix (Arize)	Traces, spans, production monitoring	Open source; OpenTelemetry-native; strong observability + eval pairing	Apache 2.0	Production monitoring + eval in one view
OpenAI Evals (hosted)	—	Shutting down. Read-only 31 Oct 2026; removed 30 Nov 2026.	—	Migrate to Promptfoo

Decision Guide

Match the tool to the question you're trying to answer:

"Is my RAG retrieval quality good?"

RAGAS → DeepEval

"Did my agent follow the right tool sequence?"

LangSmith (trajectory) · Braintrust · W&B Weave

"Can my agent be prompt-injected?"

Promptfoo (red-team mode)

"I need traces to stay on-prem"

Langfuse (self-hosted)

"I'm regulated — need audit-ready safety evidence"

Inspect (UK AISI) + NIST AI RMF documentation

"My ontology changes monthly — I need a regression harness"

DeepEval in CI/CD + ontology-derived test cases (SHACL → pytest)

"I'm on Azure / MLflow already"

TruLens (MLflow-native scorers)

"I want versioned A/B of prompt changes"

Braintrust

OpenTelemetry as the Common Layer

All Generation 2 tools — LangSmith, W&B Weave, TruLens, Langfuse, Phoenix — ingest OpenTelemetry (OTel) spans. This means you can swap the eval frontend without re-instrumenting your agent. The emerging pattern is: instrument once with OTel → fan out to multiple eval backends. The agentevals package (LangChain, not OpenAI) implements trajectory matching evaluators on top of OTel spans and works with any OpenTelemetry-instrumented framework using OpenAI models (e.g. o3-mini) as judges.

References

agentevals (LangChain) ↗ DeepEval ↗ Braintrust ↗ TruLens ↗ Inspect (UK AISI) ↗ Langfuse ↗ Promptfoo ↗ OpenAI AgentKit announcement ↗ CI/CD for AI agents 2026 ↗

AI Labs — Evaluation Approaches

Anthropic

SWE-bench Verified: Claude Opus 4.6 = 80.84% (Feb 2026, avg over 25 trials). Claude Opus 5 = 97.00% (Sep 2026, Vals.ai leaderboard).

τ-bench (tau-bench): agentic task completion — agent interacts with simulated users and APIs in customer-service scenarios. Extended as τ²-bench (dual-control, 2025).

System cards: Detailed methodology notes (e.g. averaging over 25 trials, multimodal, multilingual evals). Constitutional AI underpins internal safety eval structure.

Benchmark saturation debate: OpenAI argued Feb 2026 that SWE-bench Verified no longer measures frontier coding capability due to contamination. Anthropic methodology notes acknowledge this by averaging over many trials.

Google / DeepMind

GAIA: 466 human-annotated tasks mixing text with attached files. Accuracy: 74.5% (2026 AI Index). Extended as Gaia2 — agents run against dynamic environment events that change world state during the task.

AgentBench: evaluates agents across 8 environments — OS, Database, Knowledge Graph, Digital Card Game, Lateral Thinking, House-Holding, Web Shopping, Web Browsing.

Vertex AI Agent Engine: trajectory evaluation is first-class. Final response eval + tool-call sequence eval both native. 2026 vendor comparison: Gemini 2.5 Pro reached 100% pass rate on a task suite at lower cost than Claude Managed Agents (slowest wall time).

Less publicly documented than Anthropic's system cards. Primary references are BigBench (older) and internal red-teaming.

OpenAI

Hosted Evals — shutting down. Read-only 31 Oct 2026, removed 30 Nov 2026. Official migration: Promptfoo.

openai/evals repo: continues as community benchmark registry and LLM eval framework. Not a trajectory evaluator; no CI/CD runner shipped.

agentevals (LangChain): trajectory matching evaluators using OpenAI models (o3-mini) as judges. create_trajectory_match_evaluator — most-used pattern for OpenAI Agents SDK testing.

AgentKit + RFT (2025): Reinforcement Fine-Tuning for agents. Limited public detail on built-in testing tooling beyond what is in the API.

OpenAI's own platform eval tooling is being deprecated. The OpenAI-native testing path is: OTel traces → agentevals → Promptfoo for security.

Standards & Governance

Standard / Body	Scope	Relevance to agent testing	Status
NIST AI RMF 1.0 + AI 600-1 (Gen AI Profile)	US — AI risk management	MEASURE function covers testing methodology. AI 600-1 is the agents-specific companion. Most common governance framework cited in financial services.	Live
NIST Agent Standards Initiative	US — agent security, identity, interoperability	Directly targets AI agents. Focuses on agent identity, trust, and interoperability — where eval evidence must land.	Active (Feb 2026)
ISO/IEC 42001	International — certifiable AI management	38 Annex A controls; audit-ready. Continuous testing evidence (eval runs, CI logs) feeds directly into the MEASURE and MANAGE controls. Used as the audit wrapper over NIST RMF.	Live
Inspect (UK AISI)	UK government — safety eval harness	MIT-licensed Python framework. METR, Apollo Research, and government bodies run on it. METR migrated from Vivaria onto Inspect (Jun 2026). Inspect Hawk = AWS infrastructure layer.	Live · Government standard
METR Time Horizon 1.1	Model capability assessment	Post-2023 capability doubling time = 130.8 days (Jan 2026 report). Frontier risk reports with agent capability assessments. Sets the context for what "safe" means over time.	Active
CSA Agentic Governance v1	Cloud Security Alliance	Maps NIST AI Agent Standards to agentic governance controls. Published Mar 2026.	Published

Regulatory & Enterprise Context

EU AI Act — High-Risk Classification

Agents used in recruitment, credit, education, law enforcement, critical infrastructure, and healthcare are classified high-risk. Conformity assessment required before deployment — ISO 42001 is the most common certification path. Continuous monitoring evidence (eval run logs, CI pass rates) is a conformity artefact, not optional documentation.

SR 11-7 Extension to AI Agents (US Financial Services)

The Federal Reserve's SR 11-7 (Model Risk Management, 2011) is being extended to AI agents. An agent that makes a credit, pricing, or reserving decision must have documented validation evidence equivalent to a model validation — including out-of-sample testing and ongoing monitoring. Trajectory scoring provides this evidence; output scoring alone does not.

Academic Work — Ontology-Grounded Testing (arXiv 2606.04037)

Proposes ontology-grounded simulation and trust certification as the pre-deployment verification method for enterprise AI agents. Key argument: no conformity-assessment standard yet exists for enterprise agents. The closest analogue is pharmaceutical clinical trial structure — phased, evidence-graded, independent review. Each MVO-based agent needs a formal trust certificate before it can make production decisions.

Compositional Policy Violations (arXiv 2609.18820)

Step-level compliance can pass while workflow-level compliance fails. An agent whose every tool call passes validation can still violate a business rule at the workflow level. Testing each agent in isolation is necessary but not sufficient — the workflow composition must also be tested. Critical for any multi-step, multi-agent orchestration.

The practical stack for an enterprise AI agent in 2026: Instrument with OpenTelemetry → DeepEval for output + task completion scoring in CI → LangSmith or W&B Weave for trajectory review → ontology constraints auto-generate trap questions → Langfuse (self-hosted) where data cannot leave the tenant → ISO 42001 audit trail from CI logs → NIST AI RMF MEASURE function for the governance layer.

References

arXiv 2606.04037 — Ontology-grounded trust ↗ arXiv 2609.18820 — Compositional policy violations ↗ TRAIL — Trace Reasoning ↗ METR Frontier Risk Report May 2026 ↗ Vertex AI Agent Evaluation ↗ Claude Opus 5 System Card ↗ SWE-bench Leaderboard ↗ CSA Agentic Governance v1 ↗