AI Agent Evaluation in Production: Traces to Test Suites

Viacheslav Dubrov

2026-06-10 · Updated 2026-08-16 · 21 min · AI Agents

A final answer can say that a refund is complete while its trace shows that verify_identity never ran, issue_refund retried 17 times, or the agent declared success before the database changed. Answer-only grading hides those failures.

For engineers operating tool-using agents in production, the fix is to turn repeatable traces into bounded regression cases: deterministic checks enforce tool order, arguments, loops, and invariants; calibrated judges handle the decisions that require interpretation. The result is a versioned suite that catches the same failure before the next release.

For the short tool comparison, see Best AI Agent Evaluation Tools.

Why agent evals are different

Traditional LLM evals usually score one input-output pair: relevance, faithfulness, correctness, safety, maybe style. Agents add planning, tool calls, retries, and termination checks, and each step is a new place to fail.

Take a refund agent. The transcript can end well while the trace is wrong:

lookup_order -> issue_refund -> final_answer

The output eval passes. A trajectory eval should fail because verify_identity never ran before issue_refund. For tool-using agents, answer-only evals are smoke tests: they catch total breakage and miss everything else.

There’s a second problem: errors compound. If a workflow has 20 required steps, each succeeds independently, and every step has the same 95% reliability, its end-to-end success rate lands around 36%:

So the agent can look solid in isolated checks and still fail most full runs. The break is usually somewhere in the middle, and finding it takes component-level visibility, not another look at the answer.

Two research teams put numbers on this.

tau-bench gives an agent airline and retail customer-service tasks. The agent talks to a simulated user, calls APIs, and must follow domain policy. After the conversation, the grader checks whether the database reached the annotated goal state. A plausible transcript with the wrong rows still fails.

Under that grading GPT-4o solved only 35.2% of the airline tasks, and just above 60% of the retail ones. The paper also introduced pass^k: run the same task k times and count a pass only if the agent succeeds in all k runs.

Retail, the easier split, dropped below 25% at k = 8. On more than three quarters of those tasks, the same agent facing the same task eight times failed at least once. A one-run eval cannot expose that inconsistency.

MAST studies why agents fail. The authors built a 14-mode taxonomy from 150 hand-annotated traces, then applied it across more than 1,600 traces from 7 popular multi-agent frameworks. The taxonomy includes vague role definitions (system design), one agent ignoring what another agent reported (inter-agent misalignment), and declaring success without checking the result (no verification). These failures implicate prompts, orchestration logic, and missing checks in the harness. A stronger base model cannot execute a verification step that was never built, so the evaluation target must include the harness around the model.

The adoption gap

LangChain’s State of Agent Engineering survey (1,340 respondents, fielded in late 2025) suggests that many teams already have the raw material for better evals. It reports that 89% had some observability, 52.4% ran offline evals, and 37.3% ran online evals.

The survey also reports that 57.3% of respondents already have agents in production. When asked what blocks production, 32% named quality and 20% named latency. This is a vendor survey of its respondents, not a census of agent teams, but it exposes a useful gap between trace collection and systematic evaluation.

That leaves teams in an awkward middle state: they can inspect a bad run after the fact, then still ship the same failure twice.

Every diagnosed production failure should leave behind a trace, a label, a dataset row, and a scorer. A repeatable failure belongs in the regression suite.

Pick metrics by failure mode

The right metric depends on the failure mode, not on the framework. The useful split has three levels:

1. Outcome evals answer whether the task succeeded.
2. Trajectory evals answer whether the path was valid, efficient, and policy-compliant.
3. Component evals answer which tool, retriever, sub-agent, or decision step broke.

Each level can run offline on fixed, replayable cases before release or online on sampled production traces after the response. The guardrails section below covers that split in detail. Offline evals can require goldens: stored cases that pair an input with the outcome, tool invariants, and arguments a correct run must produce. Online evals should prefer invariants, distributions, and async checks that stay out of the request path.

Start with deterministic metrics. They’re cheap, fast, and they don’t drift.

Tool-call correctness

Tool correctness compares the called tools with the expected tools. Pick the strictness deliberately:

• Exact match: the sequence must match exactly. Use this when order is policy, for example lookup_order -> verify_identity -> issue_refund.
• In-order match: required tools must appear in the correct relative order, but extra harmless calls are allowed.
• Any-order match: required tools must appear, but order can vary.

A small local scorer is enough to start:

from collections import Counter

def tool_correctness(called: list[str], expected: list[str], mode: str = "in_order") -> float:
    if not expected:
        return 1.0
    if mode == "exact":
        return float(called == expected)
    if mode == "any_order":
        matched = sum((Counter(called) & Counter(expected)).values())
        return matched / len(expected)

    rows = [[0] * (len(expected) + 1) for _ in range(len(called) + 1)]
    for i, tool in enumerate(called):
        for j, wanted in enumerate(expected):
            if tool == wanted:
                rows[i + 1][j + 1] = rows[i][j] + 1
            else:
                rows[i + 1][j + 1] = max(rows[i][j + 1], rows[i + 1][j])
    return rows[-1][-1] / len(expected)

called = ["lookup_order", "check_refund_policy", "issue_refund"]
expected = ["lookup_order", "verify_identity", "issue_refund"]

print(round(tool_correctness(called, expected, "exact"), 3))     # 0.0
print(round(tool_correctness(called, expected, "in_order"), 3))  # 0.667

The in_order score is longest-common-subsequence recall: what fraction of the required sequence survived, in the right order. Notice what it ignores. Junk calls don’t lower it, so an agent can score 1.0 here while making twice the calls it needed. When extra calls cost money or mutate state, track precision alongside it (matched required calls over total calls) and read the two together. Recall catches the missing step; precision catches the wandering.

DeepEval’s Tool Correctness metric exposes the same knobs through should_consider_ordering and should_exact_match.

Argument correctness

Calling the right tool with the wrong arguments is often worse than calling the wrong tool because the trace looks normal.

For simple cases, validate JSON schema and exact values. For semantic cases, store expected arguments and grade the deltas:

{
    "trace_id": "tr_2417",
    "input": "Reschedule order A-100 for next Friday.",
    "expected_tools": ["lookup_order", "reschedule_delivery"],
    "expected_arguments": {
        "reschedule_delivery": {
            "order_id": "A-100",
            "date": "2026-06-19"
        }
    }
}

A tool-name metric can’t catch 2026-06-17 where the policy requires 2026-06-19. The dataset has to store arguments too.

The score that goes with that dataset is parameter-match: the fraction of expected (tool, key, value) triples the agent got right.

def argument_correctness(called_args: dict, expected_args: dict) -> float:
    total = matched = 0
    for tool, params in expected_args.items():
        for key, want in params.items():
            total += 1
            if called_args.get(tool, {}).get(key) == want:
                matched += 1
    return matched / total if total else 1.0

Exact equality is right for IDs, enums, and dates already normalized to one format. It’s wrong for free text, floats, and dates in whatever shape the model produced, where == flags a correct answer as wrong. Grade those fields on their own terms: a normalized string match, a date parse, a numeric tolerance. The metric stays the same; the per-field comparator changes.

Efficiency, loops, and dead ends

An agent that completes the task after five redundant tool calls still signals a planning problem and costs more to run.

Cheap signals you should start with:

• Redundant-call rate: identical tool calls with identical arguments repeated more than twice.
• Trace shape anomalies: sudden spikes in depth, tool-call count, token count, latency, or cost.
• Path convergence: how close the run is to the shortest known valid path for the task.
• Termination correctness: whether the agent stopped early, kept working after success, or declared success without the required state change.
• Plan adherence: if the agent writes a plan before acting, check whether the trace followed it. A good plan ignored and a bad plan followed perfectly both fail, for opposite reasons, and the diff between plan and trace tells you which.

Run these before a judge whenever you can. A loop detector is a few lines over the trace. It doesn’t need a model.

Task completion and outcome grading

Judged on the outcome, the question is “did the user get what they asked for?”

Two patterns work best:

• Referenceless task-completion judging: extract the goal from the input and judge whether the trace plus final answer achieved it. This works online because production traffic rarely has golden outputs.
• Environment-state grading: compare the final database rows, files, tickets, bookings, or records to an annotated goal state. This is more robust than transcript matching because agents can find valid paths you didn’t write down.

The second option is better when you can build it. The final state is the contract. The transcript is only evidence.

Two caveats keep this honest. A 2025 audit of agentic benchmarks found that tau-bench grades some tasks purely on the database state. On some tasks, the annotated outcome requires no state change and no specific text. An agent that does nothing can then score a pass: 38% on the airline split and 6.0% on retail, at any k. Anthropic reported an Opus 4.5 run that “failed” a booking task in tau2-bench, the successor benchmark. The agent found a policy loophole that was actually the better outcome for the user. State grading beats transcript matching, but the goal state is still an annotation, and annotations have bugs. Audit the cases that pass too easily, not only the ones that fail.

Component evals

Outcome and trajectory metrics tell you the run failed and roughly where. Component evals score one span: was the retrieved chunk relevant, did the sub-agent return the schema its caller expected, did the tool’s own response parse. Attach the score to the span rather than to the run, so “which tool degraded this week” is a query instead of a re-run.

Three checks cover most of it:

• Per-span scoring: run the metric that fits the span type. Retrieval spans get recall and precision against the annotated chunk, sub-agent spans get schema validation plus their own tool-correctness score, tool spans get error rate and latency.
• Tool-result interpretation: feed the agent a correct-but-awkward tool output (an empty list, a partial match, a stale timestamp) and check what it does next. A tool can be right while the agent reads it wrong, and that failure surfaces two steps later.
• Failure attribution: the visible failure is usually downstream of the real one. Attribute to the earliest span whose output was already wrong, not to the step that raised the error.

This is also where the compounding math from the opening pays off. If 20 steps each look fine in isolation, the run can still fail most of the time. Per-span pass rates show which step is running at 95% and which one is running at 70%.

The trace-to-eval flywheel

Mine production failures before brainstorming additional eval cases.

The loop:

1. Capture the full trace.
2. Label what failed.
3. Group similar failures.
4. Keep one representative golden per cluster.
5. Version the dataset.
6. Run it in CI.
7. Keep scoring sampled production traces online.

The companion repository trace2evals implements the full loop for a faulty support agent. It captures OpenTelemetry GenAI spans, detects failures with deterministic rules, deduplicates cases into a versioned golden dataset, and reruns each golden in CI. The default backend replaces the model with deterministic rules that re-enact the buggy agent’s decisions, so make demo reproduces the whole loop offline with no API key. Run uv sync --extra live and set an API key, and the same commands drive a real model instead.

Mine failures with error analysis

Hamel Husain and Shreya Shankar teach an error-analysis workflow for exactly this step; Hamel’s field guide walks through it. The first two steps borrow their names from qualitative research, but the method is straightforward: read traces, take notes, name the patterns.

1. Open coding: read 30 to 50 real traces and write freeform notes on what went wrong.
2. Axial coding: cluster those notes into 5 or 6 named failure categories.
3. Label everything against the taxonomy.
4. Build metrics for the largest buckets.

Don’t start with labels like reasoning_issue or tool_problem. They’re too vague to test. Use labels like missing_identity_verification, date_argument_mismatch, retried_same_tool_after_429, or stopped_before_database_update. A label that specific tells you exactly what the regression test should assert.

Deduplicate before you promote

The trace-mining loop has a trap: adding every bad trace forever. That creates a dataset that is large, expensive, and narrow. It passes on near-duplicates from March while missing the new shape of the same bug in June.

Group first. Promote one representative golden per cluster. Store the related trace IDs in metadata so a reviewer can inspect the production evidence later.

If a failure cluster recurs after a fix, the regression case did not generalize. Re-cluster and broaden the golden instead of adding 15 point examples.

Version the dataset

Version datasets the way you version prompts and code. Whenever anything meaningful changes (model, prompt, tool schema, judge prompt, or app behavior), you want to run the same dataset version before and after.

The CI gate should pin:

• dataset version
• app version
• prompt version
• judge model
• judge prompt
• evaluator code version

If any of those moves, your before/after comparison gets muddy. A goldens-v3.json file in git is fine at small scale. Tool-native snapshots in Langfuse, Phoenix, Braintrust, or LangSmith help once the dataset becomes collaborative.

Gate CI

A failing metric must become a failing build, or the eval suite is just a dashboard nobody reads.

The test should rerun the current agent against the golden input. It shouldn’t merely replay the old failed trace (sketch; the runnable version lives in the companion repository):

@pytest.mark.parametrize("golden", GOLDENS, ids=[item["id"] for item in GOLDENS])
def test_agent_regression(golden: dict) -> None:
    answer, fresh_trace = run_agent_and_capture_trace(golden["input"])

    refired = set(flag_failures(fresh_trace)) & set(golden["failure_modes"])
    assert not refired, f"failure mode regressed: {sorted(refired)}"

    assert tool_correctness(
        called=[call["name"] for call in fresh_trace["tool_calls"]],
        expected=golden["expected_tools"],
        mode=golden.get("tool_match", "in_order"),
    ) >= golden.get("tool_threshold", 1.0)

This distinction is easy to get wrong. The dataset’s job is to catch the next version of the agent repeating an old failure, not to archive the failure itself.

Calibrate the judge before trusting it

LLM-as-judge helps. It’s also easy to fool yourself with.

G-Eval evaluates three meta-evaluation benchmarks. They are SummEval, built from CNN/DailyMail news summarization; Topical-Chat, a knowledge-grounded dialogue benchmark; and QAGS, which tests factual consistency on CNN/DailyMail and XSum summaries. Using GPT-4 as the backbone, G-Eval-4 reached a Spearman correlation of 0.514 with human judgments on SummEval. Its scoring function weights rating levels by token probability ($\text{score} = \sum_i p(s_i)\,s_i$).

The paper estimated GPT-4’s token probabilities by sampling 20 times because that model did not expose them in the experiment. A hosted model may expose no usable logprobs, so keep the rubric but do not imply that you reproduced the paper’s probability weighting. These results compare the paper’s protocol with its NLG baselines on those benchmarks. They support testing an explicit rubric judge, not a general replacement for automatic metrics or a production-agent trajectory benchmark.

MT-Bench showed GPT-4 agreeing with human preferences about as often as humans agree with each other. That result helped make LLM judging mainstream. Later work exposed position, length, and self-preference biases. Judge scores can also shift when the prompt or model version changes.

JudgeBench built response pairs where one answer is objectively wrong across verifiable knowledge, reasoning, math, and code. With a plain judge prompt, GPT-4o scored 50.9%, barely above a coin flip; the paper’s stronger Arena-Hard prompt lifted the same model only to 56.6%. Swapping the model under that stronger prompt matters more: Claude 3.5 Sonnet, the best general-purpose judge tested, reached 64.3%, and o3-mini at high reasoning effort reached 80.9%. Confident but wrong answers stay hard for a judge that does not reason before it grades.

Treat the judge as a measurement instrument: calibrate it against human labels before it grades anything, and recheck it whenever the judge model or prompt moves.

When a judge is required, make the verdict structured. Schema-Guided Reasoning (SGR) gives the verdict a schema for its output shape and inspectability. Structured Outputs or constrained decoding can enforce object shape, required fields, and value constraints for fields such as evidence, passed_criteria, failed_criteria, failure_mode, and score.

Put evidence fields before the score if that makes the record easier to inspect. Field order is presentation, not a reasoning guarantee. A schema-valid verdict can still contain unsupported evidence or an unreliable score. Use calibration against human labels, deterministic validators, and transcript review to test judge reliability. CI can diff a stable JSON object, but that checks inspectability and shape rather than proving that rubric stages were followed.

A structured verdict can also change the cost curve. Treat a cheaper model as a candidate, not an automatic replacement. Run it over the same human-labeled calibration set. Compare its agreement, false-pass rate, and false-fail rate with the larger judge. Use it for routine cases only if it clears the thresholds your application set. Keep the larger judge for disagreements, high-risk cases, or calibration runs.

Default judge hygiene checklist:

1. Prefer binary pass/fail where possible. Five-point scales invite fake precision.
2. Hand-label 30 to 50 trajectories before writing the final rubric.
3. Measure judge-human agreement with Cohen’s kappa, a confusion matrix, and positive/negative recall. A judge that always says “pass” has no useful discrimination; kappa may be zero or undefined when the expected-agreement denominator is zero. Set an explicit policy for undefined kappa before using the metric as a deployment gate.
4. Decompose coarse criteria. “Did the agent verify identity before the refund tool call?” beats “Was the trajectory good?”
5. Emit the verdict through an SGR schema with evidence, failed criteria, failure mode, and score.
6. Use a judge from a different model family than the generator when possible.
7. Randomize pairwise order and average both directions.
8. Penalize unsupported length in the rubric. A longer answer is not a better one.
9. Pin the judge model, prompt, dataset, schema, and app version.
10. Recalibrate after model, prompt, tool, policy, or schema changes.

For high-stakes scores, use a small jury instead of one big judge. PoLL tested a panel of smaller judges drawn from disjoint model families and pooled their verdicts. Across six datasets, the panel tracked human judgments better than a single GPT-4 judge. It also avoided the single judge’s self-preference bias and cost over seven times less. Keep human review for decisions that affect money, access, safety, or compliance.

If a judge agrees with humans at 0.55 kappa on your task, don’t use it to block deploys. Use it to sort review queues. If it sits near 0.75 and the failure cost is moderate, a CI gate is much easier to defend.

Guardrails block inline, online evals observe afterward

People mix these up because both produce scores. The difference is placement: inline in the request path, before release, or after the response.

Guardrails run inline. They are fast and user-visible. A guardrail can block a tool call, redact PII, reject prompt injection, or force a retry before the response leaves your system. A false positive is a production bug. A false negative is quieter and worse because nothing in the request path reports it. Schema, range, and policy checks are deterministic. Injection and PII detection are classifiers, so treat misses as expected and keep an async eval watching what they let through.

Offline evals run before release. They are reproducible. They gate prompts, models, tools, retrievers, and policies against a fixed dataset.

Online evals run after the response, usually on sampled traffic. They can use slower LLM judges because they are not in the latency path. Their job is to detect drift, find new failure clusters, and feed the next offline dataset.

Get the placement wrong and it hurts either way:

• A judge in the request path adds latency and a new source of flakiness.
• A guardrail relegated to async scoring lets policy violations reach users.

For high-volume systems, score a small sample with a stronger judge and a wider sample with cheaper classifiers. Alert on clusters and confidence bounds, not one noisy point estimate.

Tooling choices

No single tool owns the whole loop. Compare a trace/dataset store and a CI/eval runner separately; one product can cover both, but you do not need to buy both from one vendor.

This is an author snapshot checked on 2026-08-16. Each link is the current documentation I used for the capability claim. Plans, licenses, API keys, provider access, and infrastructure requirements still apply.

The trade-off notes describe where cost comes from, not what it is. Pricing pages move, and vendors count different things: traces, observations, spans, scores, users, retention, or processed data. Recheck live pricing before committing.

Recommendations by constraint:

• Choose Phoenix when self-hosting, privacy, and OTel-compatible tracing are hard requirements and your team can operate the deployment.
• Choose Langfuse when you also need dataset versioning and experiments, and you can operate its storage stack or buy the required add-ons.
• Choose DeepEval when Python/pytest CI pass-fail is the primary contract.
• Choose Inspect AI when the main work is safety or frontier-agent evaluation in configurable sandboxes.
• Choose LangSmith when LangChain/LangGraph integration outweighs the Enterprise requirement for hybrid or self-hosted deployment.
• Choose Braintrust when managed pull-request feedback and experiment comparison justify an API-key-backed service.
• Choose Promptfoo when prompt or red-team checks are the main regression surface and a trace store is out of scope.

Tool choice is secondary. If production failures don’t become test cases, you’re mostly paying for trace storage.

A practical rollout checklist

Build the evidence pipeline before expanding the metric stack. Start by deciding where the examples will come from.

1. Collect historical runs first. If the agent already exists, pull traces, support tickets, bug reports, thumbs-down sessions, manual QA transcripts, and dogfooding notes before changing the implementation. If the agent does not exist yet, log every prototype and manual test run from day one.

2. Instrument the trace shape. Capture messages, tool calls, arguments, tool outputs, errors, token counts, latency, cost, user feedback, app version, prompt version, model version, tool schema version, and final environment state. Use OpenTelemetry GenAI conventions or OpenInference-style spans if you want portability. Use Langfuse, LangSmith, Phoenix, or Braintrust if you want a trace UI and dataset workflow immediately.

3. Turn real failures into seed cases. Read the traces before summarizing them with a model. For each useful failure, store the input, source trace ID, expected state, expected tool invariants, failure mode, severity, and reviewer note. Langfuse can link dataset items back to production traces; LangSmith can create datasets from traced runs. Keep the source link so the case remains auditable.

4. If there is no history, generate cold-start cases. Ask an LLM to draft tasks from product requirements, policies, tool schemas, state machines, and support macros. Cover happy paths and failures such as wrong permissions, missing identity checks, stale tool results, ambiguous dates, retries after rate limits, and contradictory tool output.

5. Do not trust synthetic cases until a human reviews them. Synthetic examples are useful for coverage, not truth. Mark them with source: synthetic and require a reviewer to approve the expected outcome. Run a known-good reference path when possible, and use different model families to generate the case and judge the result.

6. Build a small balanced dataset. Include successes, failures, refusals, boundary cases, long-turn cases, policy-sensitive cases, and valid alternate paths. Do not make the golden “the exact old transcript.” Store what a golden stores above, plus the failure mode that put the case in the suite.

7. Add deterministic checks first. Required tool order where order is policy, required arguments, schema validation, final-state diffs, loop limits, token and latency ceilings, and task-specific invariants should run before any judge.

8. Add one SGR-shaped judge. Use it only for the part that needs interpretation. Calibrate it against human labels. If it cannot separate good and bad examples on the calibration set, fix the rubric before wiring it into CI.

9. Wire the loop. Run the small offline suite in CI, run the larger suite before release, score sampled production traffic online, and promote recurring online failure clusters back into the offline dataset.

Your first eval suite will be wrong in boring ways. Ship it anyway. A suite you run every day is easier to fix than a perfect design doc that never blocks a bad PR.

References

• trace2evals companion demo repository
• LangChain - State of Agent Engineering
• Anthropic - Demystifying Evals for AI Agents
• Hamel Husain - A Field Guide to Rapidly Improving AI Products
• OpenTelemetry semantic conventions for generative AI systems
• DeepEval Tool Correctness
• DeepEval Task Completion
• DeepEval unit testing in CI/CD
• Inspect running evals
• OpenAI Structured model outputs
• Schema-Guided Reasoning: vLLM, xgrammar, and Pydantic
• Phoenix self-hosting
• Langfuse datasets and versioning
• Langfuse experiments in CI/CD
• LangSmith platform setup
• LangSmith - Create and manage datasets programmatically
• Braintrust - Create experiments
• Phoenix LLM evals
• Promptfoo GitHub Action integration
• tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• Why Do Multi-Agent LLM Systems Fail?
• Establishing Best Practices for Building Rigorous Agentic Benchmarks
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• Self-Preference Bias in LLM-as-a-Judge