AI Agent Evaluation in Production: Traces to Test Suites

Viacheslav Dubrov

2026-06-10 · Updated 2026-10-02 · 27 min · AI Agents

This guide is for engineers evaluating tool-using AI agents in production. You will learn how to turn failed runs into repeatable tests, check the agent’s actions as well as its answers, and use those tests before the next release.

For the short tool comparison, see Best AI Agent Evaluation Tools.

Why agent evals are different

Traditional LLM evals usually score one input-output pair: relevance, faithfulness, correctness, safety, maybe style. Agents add planning, tool calls, retries, and termination checks, and each step is a new place to fail.

Take a refund agent. The transcript can end well while the trace is wrong:

lookup_order -> issue_refund -> final_answer

The output eval passes. A trajectory eval should fail because verify_identity never ran before issue_refund. For tool-using agents, answer-only evals can catch content-quality failures, but they cannot establish that the agent took a valid trajectory or produced the required side effects.

The left card is the row an answer-only eval reads; the right card is the trace a trajectory eval reads, with the skipped identity check.

There’s a second problem: errors compound. If a workflow has 20 required steps, each succeeds independently, and every step has the same 95% reliability, its end-to-end success rate lands around 36%:

So the agent can look solid in isolated checks and still fail most full runs. The break is usually somewhere in the middle, and finding it takes component-level visibility, not another look at the answer.

tau-bench gives an agent airline and retail customer-service tasks. The agent talks to a simulated user, calls APIs, and must follow domain policy. After the conversation, the grader checks whether the database reached the annotated goal state. A plausible transcript with the wrong rows still fails.

Under that grading GPT-4o solved only 35.2% of the airline tasks, and just above 60% of the retail ones. The paper also introduced pass^k: the chance that all k independent trials pass, averaged across tasks.

Retail, the easier split, had pass^8 below 25%. For a randomly selected retail task and eight independent trials, the chance that all eight runs passed was below 25%. A one-run eval cannot measure that consistency.

The chart plots both cases: the chance that every step or every trial passes drops with each one you add. At 90% per step the drop is steeper: if each of 20 independent steps passes 90% of the time, all 20 pass about 12% of the time (0.9^20 ≈ 0.12).

MAST studies why agents fail. The authors built a 14-mode taxonomy from 150 hand-annotated traces, then applied it across more than 1,600 traces from 7 popular multi-agent frameworks. The taxonomy includes vague role definitions (system design), one agent ignoring what another agent reported (inter-agent misalignment), and declaring success without checking the result (no verification). These failures implicate prompts, orchestration logic, and missing checks in the harness. A stronger base model cannot execute a verification step that was never built, so the evaluation target must include the harness around the model.

The adoption gap

LangChain’s State of Agent Engineering survey (1,340 respondents, fielded in late 2025) suggests that many teams already have the raw material for better evals. It reports that 89% had some observability, 52.4% ran offline evals, and 37.3% ran online evals.

The survey also reports that 57.3% of respondents already have agents in production. When asked what blocks production, 32% named quality and 20% named latency. This is a vendor survey of its respondents, not a census of agent teams, but it exposes a useful gap between trace collection and systematic evaluation.

That leaves teams in an awkward middle state: they can inspect a bad run after the fact, then still ship the same failure twice.

Every diagnosed production failure should leave behind a trace, a label, a dataset row, and a scorer. A repeatable failure belongs in the regression suite.

Pick metrics by failure mode

Choose the metric by the failure you need to catch. The useful split has three levels:

1. Outcome evals answer whether the task succeeded.
2. Trajectory evals answer whether the path was valid, efficient, and policy-compliant.
3. Component evals answer which tool, retriever, sub-agent, or decision step broke.

The figure shows which part of the same run each level reads.

Each level can run offline on fixed, replayable cases before release or online on sampled production traces after the response. The guardrails section below covers that split in detail. Offline evals can require goldens: stored cases that pair an input with the outcome, tool invariants, and arguments a correct run must produce. A full golden appears under “Deduplicate before you promote” below. Online evals should prefer invariants, distributions, and async checks that stay out of the request path.

Start with deterministic metrics. They repeat for fixed inputs and code, and cost little to run. Their rules can still become stale when tools or policy change, so version the scorer with the specification it checks.

Pick the checker by the criterion: code when code can check it, a review queue while no calibrated judge exists, a person when the verdict starts a consequential action, and a calibrated LLM judge for the rest.

Tool-call correctness

Tool correctness compares the called tools with the expected tools. Pick the strictness deliberately:

• Exact match: the sequence must match exactly. Use this when order is policy, for example lookup_order -> verify_identity -> issue_refund.
• In-order match: required tools must appear in the correct relative order, but extra harmless calls are allowed.
• Any-order match: required tools must appear, but order can vary.

The figure scores one trace under all three modes. The agent skipped verify_identity and made an extra check_refund_policy call.

A small local scorer is enough to start:

from collections import Counter

def tool_correctness(called: list[str], expected: list[str], mode: str = "in_order") -> float:
    if mode not in {"exact", "in_order", "any_order"}:
        raise ValueError(f"unknown matching mode: {mode}")
    if mode == "exact":
        return float(called == expected)
    if not expected:
        return 1.0
    if mode == "any_order":
        matched = sum((Counter(called) & Counter(expected)).values())
        return matched / len(expected)

    rows = [[0] * (len(expected) + 1) for _ in range(len(called) + 1)]
    for i, tool in enumerate(called):
        for j, wanted in enumerate(expected):
            if tool == wanted:
                rows[i + 1][j + 1] = rows[i][j] + 1
            else:
                rows[i + 1][j + 1] = max(rows[i][j + 1], rows[i + 1][j])
    return rows[-1][-1] / len(expected)

called = ["lookup_order", "check_refund_policy", "issue_refund"]
expected = ["lookup_order", "verify_identity", "issue_refund"]

print(round(tool_correctness(called, expected, "exact"), 3))     # 0.0
print(round(tool_correctness(called, expected, "in_order"), 3))  # 0.667
assert tool_correctness(["issue_refund"], [], "exact") == 0.0
assert tool_correctness([], [], "exact") == 1.0
assert tool_correctness(["a", "b"], ["a", "a", "b"], "any_order") == 2 / 3
try:
    tool_correctness([], [], "typo")
except ValueError:
    pass
else:
    raise AssertionError("unknown modes must fail")

The in_order score is longest-common-subsequence recall: what fraction of the required sequence survived, in the right order. Notice what it ignores. Junk calls don’t lower it, so an agent can score 1.0 here while making twice the calls it needed. When extra calls cost money or mutate state, track precision alongside it (matched required calls over total calls) and read the two together. Recall catches the missing step; precision catches the wandering. A good tool score does not make an extra refund or write safe. Check every state-changing call on its own: permissions, resource, arguments, and the verification that must come first. If no tools are expected, only exact mode fails a run that calls tools; the other two modes return 1.0.

DeepEval’s Tool Correctness metric exposes the same knobs through should_consider_ordering and should_exact_match.

Argument correctness

Calling the right tool with the wrong arguments is often worse than calling the wrong tool because the trace looks normal.

For simple cases, validate JSON schema and exact values. For semantic cases, store the expected arguments with the case, as the expected_arguments field of the golden under “Deduplicate before you promote” does, and grade the deltas. A tool-name metric can’t catch 2026-06-17 where the policy requires 2026-06-19.

For this one-call-per-tool illustration, parameter-match is the fraction of expected (tool, key, value) triples the agent got right. The dictionaries below are valid only when each relevant tool has at most one invocation. Do not build them by overwriting earlier calls with the same name: that would hide a wrong refund followed by a correct one. For repeated calls, retain call IDs and order, match the intended invocation, and validate every mutation separately.

def argument_correctness(called_args: dict, expected_args: dict) -> float:
    total = matched = 0
    for tool, params in expected_args.items():
        for key, want in params.items():
            total += 1
            if key in called_args.get(tool, {}) and called_args[tool][key] == want:
                matched += 1
    return matched / total if total else 1.0

assert argument_correctness({}, {"reschedule_delivery": {"date": None}}) == 0.0
assert argument_correctness({"reschedule_delivery": {}},
                            {"reschedule_delivery": {"date": None}}) == 0.0
assert argument_correctness({"reschedule_delivery": {"date": None}},
                            {"reschedule_delivery": {"date": None}}) == 1.0

Exact equality is right for IDs, enums, and dates already normalized to one format. It’s wrong for free text, floats, and dates in whatever shape the model produced, where == flags a correct answer as wrong. Grade those fields on their own terms: a normalized string match, a date parse, a numeric tolerance. The metric stays the same; the per-field comparator changes.

Efficiency, loops, and dead ends

An agent that completes the task after five redundant tool calls still signals a planning problem and costs more to run.

Cheap signals you should start with:

• Redundant-call rate: identical tool calls with identical arguments repeated more than twice.
• Trace shape anomalies: sudden spikes in depth, tool-call count, token count, latency, or cost.
• Path convergence: how close the run is to the shortest known valid path for the task.
• Termination correctness: whether the agent stopped early, kept working after success, or declared success without the required state change.
• Plan adherence: if the agent writes a plan before acting, check whether the trace followed it. A good plan ignored and a bad plan followed perfectly both fail, for opposite reasons, and the diff between plan and trace tells you which.

Run these before a judge whenever you can. A loop detector is a few lines over the trace and needs no model. This one counts the most repeated identical call in a trace:

import json

def redundant_call_count(tool_calls: list[dict]) -> int:
    seen: dict[str, int] = {}
    for call in tool_calls:
        key = call["name"] + json.dumps(call["arguments"], sort_keys=True)
        seen[key] = seen.get(key, 0) + 1
    return max(seen.values(), default=0)

calls = [{"name": "lookup_order", "arguments": {"order_id": "A-100"}}] * 3
assert redundant_call_count(calls) == 3
assert redundant_call_count([]) == 0

Flag a trace when the count is above 2. The companion repository uses the same rule.

Task completion and outcome grading

Judged on the outcome, the question is “did the user get what they asked for?”

Two patterns work best:

• Referenceless task-completion judging: extract the goal from the input and judge whether the trace plus final answer achieved it. This works online because production traffic rarely has golden outputs.
• Environment-state grading: compare the final database rows, files, tickets, bookings, or records to an annotated goal state. This is more robust than transcript matching because agents can find valid paths you didn’t write down.

The second option is better when you can build it. Grade the final state; use the transcript to explain a failure.

State grading has two limits. A 2025 audit of agentic benchmarks found that tau-bench grades some tasks purely on the database state. On some tasks, the annotated outcome requires no state change and no specific text. An agent that does nothing can then score a pass: 38% on the airline split and 6.0% on retail, at any k. Anthropic reported an Opus 4.5 run that “failed” a booking task in tau2-bench, the successor benchmark. The agent found a policy loophole that gave the user a better outcome. State grading beats transcript matching, but the goal state is still an annotation, and annotations have bugs. Audit the cases that pass too easily, not only the ones that fail.

Component evals

Outcome and trajectory metrics tell you the run failed and roughly where. Component evals score one span: was the retrieved chunk relevant, did the sub-agent return the schema its caller expected, did the tool’s own response parse. Attach the score to the span rather than to the run, so “which tool degraded this week” is a query instead of a re-run.

Three checks cover most of it:

• Per-span scoring: run the metric that fits the span type. Retrieval spans get recall and precision against the annotated chunk, sub-agent spans get schema validation plus their own tool-correctness score, tool spans get error rate and latency.
• Tool-result interpretation: feed the agent a correct-but-awkward tool output (an empty list, a partial match, a stale timestamp) and check what it does next. A tool can be right while the agent reads it wrong, and that failure surfaces two steps later.
• Failure attribution: the visible failure is usually downstream of the real one. Attribute to the earliest span whose output was already wrong, not to the step that raised the error.

The figure shows a five-span refund run. The agent plans to find the order and refund it. search_orders returns an old order ID, lookup_order fetches that wrong order, issue_refund raises an error because the order cannot be refunded, and final_answer tells the user the refund failed. The error appears at issue_refund (step 4), but the first wrong output came from search_orders (step 2), two spans earlier.

This is where the 20-step calculation from the opening applies. If 20 steps each look fine in isolation, the run can still fail most of the time. Per-span pass rates show which step is running at 95% and which one is running at 70%.

The trace-to-eval flywheel

Mine production failures before brainstorming additional eval cases. Each numbered step in the figure matches the list below.

The loop:

1. Capture enough trace evidence to reconstruct the failure, with sensitive content controlled.
2. Label what failed.
3. Group similar failures.
4. Keep representative goldens, including variants that need different outcomes. A reviewer approves each expected outcome.
5. Version the dataset.
6. Run it in CI.
7. Keep scoring sampled production traces online.

The companion repository trace2evals demonstrates this workflow with a faulty support agent. It records tool calls and shop state in local OpenTelemetry JSONL, detects failures with deterministic rules, and emits versioned review candidates. A reviewer must define and approve their expected outcomes before using them as release tests.

Run uv sync --locked and make demo in that repository. No API key is needed: a scripted agent repeats planted mistakes. The demo records six runs, finds four failed requests, and writes them as review candidates. It then runs ten reference tasks in three configurations. The buggy v1 agent fails five. The fixed v2 agent passes all ten, both when the tools check permissions and when only the prompt asks for checks. The reports are data/demo/red.json, prompt-only.json, and green.json. The README explains the optional live run with an OpenAI model.

The demo uses synthetic cases, in-memory tools, and a repository-specific trace schema; its adapter does not accept arbitrary OpenTelemetry exports.

Mine failures with error analysis

Hamel Husain and Shreya Shankar teach an error-analysis workflow for this step; Hamel’s field guide walks through it: inspect real conversations, take open-ended notes, categorize failures, and build specific tests. The first two steps take their names from qualitative research, which helps when you look for the method elsewhere.

1. Open coding: inspect traces and write open-ended notes on what went wrong.
2. Axial coding: group the notes into named failure categories.
3. Label traces against that taxonomy.
4. Build specific tests for the largest actionable clusters.

Don’t start with labels like reasoning_issue or tool_problem. They’re too vague to test. Use labels like missing_identity_verification, date_argument_mismatch, retried_same_tool_after_429, or stopped_before_database_update. A label that specific tells you exactly what the regression test should assert.

Deduplicate before you promote

A common mistake is adding every bad trace to the suite. That creates a dataset that is large, expensive, and narrow. It passes on near-duplicates from March while missing the new shape of the same bug in June.

Group first. Start with one representative golden per cluster, then retain variants with different permissions, arguments, recovery states, or expected outcomes. Similar wording does not make two policy cases equivalent. Store related trace IDs in access-controlled metadata so a reviewer can inspect the evidence later.

If a failure cluster recurs after a fix, the regression case did not generalize. Revisit the cluster and add the missing behavioral variants rather than collecting near-identical transcripts.

A promoted golden stores the input, the source traces, the set it belongs to, the failure label that created it, the expected tools, arguments, and final state, and who approved it. These are the fields the companion repository’s promotion steps use:

{
    "id": "reschedule-date",
    "input": "Reschedule order A-100 for June 19, 2026.",
    "source_trace_ids": ["tr_2417"],
    "split": "regression",
    "failure_modes": ["date_argument_mismatch"],
    "expected_tools": ["lookup_order", "reschedule_delivery"],
    "tool_match": "in_order",
    "expected_arguments": {
        "reschedule_delivery": { "order_id": "A-100", "date": "2026-06-19" }
    },
    "expected_state": {
        "reschedules": [{ "order_id": "A-100", "date": "2026-06-19" }],
        "refunds": []
    },
    "review_status": "approved",
    "reviewed_by": "support-policy-owner"
}

expected_state lets the test grade the shop after the run. split keeps the case out of held-out sets, which the next section covers.

Version the dataset

Version datasets the way you version prompts and code. Whenever anything meaningful changes (model, prompt, tool schema, judge prompt, or app behavior), you want to run the same dataset version before and after.

The CI check should pin:

• dataset version
• app version
• prompt version
• judge model
• judge prompt
• evaluator code version
• tool schemas, policy, harness and context-management configuration
• model revision, reasoning and sampling settings
• initial environment fixture and permitted external effects

If any of those moves, you cannot tell which change caused the difference. A goldens-v3.json file in git is fine at small scale. Tool-native snapshots in Langfuse, Phoenix, Braintrust, or LangSmith help once the dataset becomes collaborative.

Keep development regressions, judge-calibration examples, held-out validation, and monitoring samples separate. Group related sessions, users, and tasks before splitting so near-duplicates cannot leak across sets. Once a case shapes a prompt or rubric, treat it as development evidence. A mined failure suite tests known regressions; its average does not estimate the production success rate.

Reset mutable state for every trial: files, database rows, caches, and tool fixtures. Isolate credentials and external effects, and keep candidate and baseline budgets equal. Report the number of tasks separately from trials, along with all started attempts, timeouts, crashes, and unscorable outcomes. Compare paired results on the same tasks. These choices follow the clean-trial and outcome-grading approach described in Anthropic’s evaluation guide.

Pin benchmark and sandbox versions

The tau-bench numbers in the first section explain repeated-trial reliability; they are not a current model leaderboard. Public benchmarks change tasks and graders between releases, and scores from different grader versions are not comparable. The maintained tau-bench repository shipped such a grading fix in July 2026. Pin the task and grader revisions as well as the model, and re-score saved trajectories when an annotation changes.

Choose a benchmark that exercises the deployed interface. Text-only customer-service tests cannot establish interruption handling in a voice agent. Knowledge-retrieval tasks also need the corpus, search configuration, and evidence available at each turn. Borrow these task shapes for local regression cases rather than importing a public leaderboard rank as a release criterion.

The sandbox is part of the test too. Anthropic’s February 2026 infrastructure study found a six-percentage-point Terminal-Bench 2.0 difference between its strict and uncapped resource setups with the same model, harness, and tasks. Extra headroom both reduced infrastructure failures and enabled different solution strategies. Record CPU and RAM guarantees and limits, timeouts, concurrency, network access, and infrastructure-error handling. Report those failures separately without quietly dropping them from the expected-task denominator. The same study recommends skepticism toward leaderboard differences below three percentage points until the evaluation setup is documented and matched; apply that rule to your own comparisons as well.

Run evals in CI

A release check must fail the build when a metric crosses its agreed limit. Otherwise the eval suite is only a dashboard.

After restoring the case fixture in an isolated trial, the test runs the current agent on the golden input. It does not replay the stored failed trace, which would only show that the old run failed (sketch; the runnable version is evals/test_agent_regression.py, where grade() runs every check, not only the labels that created the case):

@pytest.mark.parametrize("golden", GOLDENS, ids=[item["id"] for item in GOLDENS])
def test_agent_regression(golden: dict) -> None:
    answer, fresh_trace = run_agent_and_capture_trace(golden["input"])

    refired = set(flag_failures(fresh_trace)) & set(golden["failure_modes"])
    assert not refired, f"failure mode regressed: {sorted(refired)}"

    assert tool_correctness(
        called=[call["name"] for call in fresh_trace["tool_calls"]],
        expected=golden["expected_tools"],
        mode=golden.get("tool_match", "in_order"),
    ) >= golden.get("tool_threshold", 1.0)

Calibrate the judge before trusting it

LLM-as-judge helps. It’s also easy to fool yourself with.

G-Eval had GPT-4 write rubric steps before scoring and reached a 0.514 Spearman correlation with human ratings on SummEval, above the older automatic metrics. It weighted each score by token probability; GPT-4 did not expose those probabilities, so the paper estimated them by sampling 20 times. Keep the rubric; do not claim the weighting unless your judge returns logprobs. These are summarization results, not agent-trajectory results.

MT-Bench showed GPT-4 agreeing with human preferences about as often as humans agree with each other. That result helped make LLM judging mainstream. Later work exposed biases that each need their own check:

JudgeBench built response pairs where one answer is objectively wrong across verifiable knowledge, reasoning, math, and code. With a plain judge prompt, GPT-4o scored 50.9%, barely above a coin flip; the paper’s stronger Arena-Hard prompt lifted the same model only to 56.6%. Swapping the model under that stronger prompt matters more: Claude 3.5 Sonnet, the best general-purpose judge tested, reached 64.3%, and o3-mini at high reasoning effort reached 80.9%. Confident but wrong answers stay hard for a judge that does not reason before it grades.

Treat the judge as a measurement instrument: calibrate it against human labels before it grades anything, and recheck it whenever the judge model or prompt moves. The figure shows the loop and its comparison step: which judge errors count against which human labels.

When a judge is required, make the verdict structured. Schema-Guided Reasoning (SGR) gives the verdict a schema for its output shape and inspectability. Structured Outputs or constrained decoding can enforce object shape, required fields, and value constraints for fields such as evidence, passed_criteria, failed_criteria, failure_mode, and passed.

A binary verdict with criterion lists looks like this; CI compares passed with the human label:

{
    "type": "object",
    "required": [
        "evidence",
        "passed_criteria",
        "failed_criteria",
        "failure_mode",
        "passed"
    ],
    "properties": {
        "evidence": { "type": "array", "items": { "type": "string" } },
        "passed_criteria": { "type": "array", "items": { "type": "string" } },
        "failed_criteria": { "type": "array", "items": { "type": "string" } },
        "failure_mode": { "type": ["string", "null"] },
        "passed": { "type": "boolean" }
    },
    "additionalProperties": false
}

Put evidence fields before the verdict if that makes the record easier to inspect. Field order is presentation, not a reasoning guarantee. A schema-valid verdict can still contain unsupported evidence or a wrong passed value. Use calibration against human labels, deterministic validators, and transcript review to test judge reliability. CI can diff a stable JSON object, but that checks inspectability and shape rather than proving that rubric stages were followed.

A structured verdict also makes a cheaper judge easier to test. Treat a cheaper model as a candidate, not an automatic replacement. Run it over the same human-labeled calibration set. Compare its agreement, false-pass rate, and false-fail rate with the larger judge. Use it for routine cases only if it clears the thresholds your application set. Keep the larger judge for disagreements, high-risk cases, or calibration runs.

Default judge hygiene checklist:

1. Prefer binary pass/fail where possible. Five-point scales invite fake precision.
2. Label trajectories covering the actual failure modes before finalizing the rubric. Choose sample size from coverage and the uncertainty the decision can tolerate, then reserve separate cases for validation.
3. Measure judge-human agreement with Cohen’s kappa, a confusion matrix, and positive/negative recall. Kappa measures agreement after accounting for agreement expected by chance; higher is better. A judge that always says “pass” has no useful discrimination, so kappa may be zero or undefined. Decide what to do when it is undefined before using the metric to approve a release.
4. Decompose coarse criteria. “Did the agent verify identity before the refund tool call?” beats “Was the trajectory good?”
5. Emit the verdict through an SGR schema with evidence, criterion lists, failure mode, and a pass/fail field.
6. Compare same- and cross-family judges against held-out human labels; family separation alone does not establish reliability.
7. Measure pairwise order sensitivity. Keep randomization or swapped-order aggregation only if it improves held-out decisions; a 2026 controlled study found that swapping could hurt on adversarial cases.
8. Give no credit for extra text unless it adds correct, relevant, supported content. A longer answer is not a better one.
9. Pin the judge model, prompt, dataset, schema, and app version.
10. Recalibrate after model, prompt, tool, policy, or schema changes.

A panel is another candidate to test. PoLL reported better alignment with human judgments, reduced intra-model bias, and lower cost than its single-GPT-4 baseline across six datasets. Those results belong to its models, tasks, and historical prices. They do not establish that a panel is safer on your task. Compare its false passes, false failures, cost, and disagreement workload with one calibrated judge on held-out labels.

There is no universal kappa threshold that makes a judge suitable for CI. Report the confusion matrix, label counts, false-pass rate among human failures, and false-fail rate among human passes, with uncertainty. Choose release limits from the consequences of those errors. Use review queues when evidence is too weak for automatic acceptance, and retain human authorization for consequential actions where the workflow requires it.

Guardrails block inline, online evals observe afterward

People mix these up because both produce scores. The difference is placement: inline in the request path, after the response, or before release. The figure puts the three placements on one time line, with what each one can stop.

Guardrails run inline. They are fast and user-visible. A guardrail can block a tool call, redact PII, reject prompt injection, or force a retry before the response leaves your system. A false positive is a production bug. A false negative is quieter and worse because nothing in the request path reports it. Schema, range, and policy checks are deterministic. Injection and PII detection are classifiers, so treat misses as expected and keep an async eval watching what they let through.

Online evals run after the response, usually on sampled traffic. They can use slower LLM judges because they are not in the latency path. Their job is to detect drift, find new failure clusters, and feed the next offline dataset.

Offline evals run before release. They are reproducible. They check prompts, models, tools, retrievers, and policies against a fixed dataset.

Each wrong placement has a cost:

• A judge in the request path adds latency and a new source of flakiness.
• A guardrail relegated to async scoring lets policy violations reach users.

For security tests, distinguish attack detection, a prohibited action being attempted, and a harmful effect actually succeeding. Report successful harmful effects per attack trial, with the threat model and attempt budget, alongside legitimate task success and false blocks per benign trial. A detector score alone cannot establish that data stayed private or that a write was prevented. Use isolated targets; Anthropic’s cybersecurity evaluation incident report documents why evaluation effects need containment.

For high-volume systems, score a small sample with a stronger judge and a wider sample with cheaper classifiers. Alert on clusters and confidence bounds, not one noisy point estimate.

Tooling choices

No single tool owns the whole loop. Compare a trace/dataset store and a CI/eval runner separately; one product can cover both, but you do not need to buy both from one vendor. The Covers column shows which roles each tool fills.

Vendors bill by different units (traces, spans, scores, seats, retention, or processed data), so compare live pricing on your own trace volume. The cost drivers above were checked against the pricing pages in October 2026.

Tool choice is secondary. If production failures don’t become test cases, you’re mostly paying for trace storage.

A practical rollout checklist

Build the evidence pipeline before expanding the metric stack. Start by deciding where the examples will come from.

1. Collect historical runs first. If the agent already exists, pull traces, support tickets, bug reports, thumbs-down sessions, manual QA transcripts, and dogfooding notes before changing the implementation. If the agent does not exist yet, log every prototype and manual test run from day one.

2. Instrument the trace shape. Capture messages, tool calls, arguments, tool outputs, errors, token counts, latency, cost, user feedback, app version, prompt version, model version, tool schema version, and final environment state. Use OpenTelemetry GenAI conventions or OpenInference-style spans if you want portability, and pin the convention version and adapter. Capture content selectively: redact secrets and personal data, restrict access, and set retention before promoting traces into datasets. Use Langfuse, LangSmith, Phoenix, or Braintrust if you want a trace UI and dataset workflow immediately.

3. Turn real failures into seed cases. Read the traces before summarizing them with a model. For each useful failure, store the input, source trace ID, expected state, expected tool invariants, failure mode, severity, and reviewer note. Langfuse can link dataset items back to production traces; LangSmith can create datasets from traced runs. Keep the source link so the case remains auditable.

4. If there is no history, generate cold-start cases. Ask an LLM to draft tasks from product requirements, policies, tool schemas, state machines, and support macros. Cover happy paths and failures such as wrong permissions, missing identity checks, stale tool results, ambiguous dates, retries after rate limits, and contradictory tool output.

5. Do not trust synthetic cases until a human reviews them. Synthetic examples are useful for coverage, not truth. Mark them with source: synthetic and require a reviewer to approve the expected outcome. Run a known-good reference path when possible, and validate generated expectations independently; using another model family does not replace that check.

6. Build a small balanced dataset. Include successes, failures, refusals, boundary cases, long-turn cases, policy-sensitive cases, and valid alternate paths. Do not make the golden “the exact old transcript.” Store the fields from the golden example under “Deduplicate before you promote”, including the failure mode that put the case in the suite.

7. Add deterministic checks first. Required tool order where order is policy, required arguments, schema validation, final-state diffs, loop limits, token and latency ceilings, and task-specific invariants should run before any judge.

8. Add one SGR-shaped judge. Use it only for the part that needs interpretation. Calibrate it against human labels and test the chosen rubric on untouched validation cases. If it cannot separate good and bad examples on the calibration set, fix the rubric before wiring it into CI.

9. Wire the loop. Run the small offline suite in CI, run the larger suite before release, score sampled production traffic online, and promote recurring online failure clusters back into the offline dataset.

Your first eval suite will miss cases. Run it anyway, then add recurring failures as cases. A suite that runs every day gives you evidence to improve it.

References

• trace2evals companion demo repository
• LangChain - State of Agent Engineering
• Anthropic - Demystifying Evals for AI Agents
• Hamel Husain - A Field Guide to Rapidly Improving AI Products
• OpenTelemetry semantic conventions for generative AI systems
• DeepEval Tool Correctness
• DeepEval Task Completion
• DeepEval unit testing in CI/CD
• Inspect running evals
• OpenAI Structured model outputs
• Schema-Guided Reasoning: vLLM, xgrammar, and Pydantic
• Phoenix self-hosting
• Langfuse datasets and versioning
• Langfuse experiments in CI/CD
• LangSmith platform setup
• LangSmith - Create and manage datasets programmatically
• Braintrust - Create experiments
• Phoenix LLM evals
• Promptfoo GitHub Action integration
• tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• Why Do Multi-Agent LLM Systems Fail?
• Establishing Best Practices for Building Rigorous Agentic Benchmarks
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• Self-Preference Bias in LLM-as-a-Judge