AI agent evaluation: Phoenix, LangSmith, and DeepEval
No agent evaluation tool can replace a test design. Start with the failures the product must catch: wrong answers, bad tool choices, invalid arguments, or unsafe side effects. Also check unnecessary steps and regressions in latency and cost. Then choose the tool that fits where those tests must run.
Use Phoenix when open tracing and self-hosting matter. Use LangSmith when datasets, traces, annotation, and experiments should share one managed workflow. Use DeepEval when evaluation must look like tests in a Python CI pipeline. Add Promptfoo for a local CLI, matrix tests, and red-team cases.
Choose by where tests run, which steps you need to inspect, and whether evaluation data must stay in your environment.
Decision table
| Need | Starting point | Why |
|---|---|---|
| OpenTelemetry traces and self-hosting | Phoenix | Open-source tracing, evaluations, datasets, and experiments use OpenTelemetry and OpenInference conventions. |
| Managed traces, datasets, and review | LangSmith | It evaluates trajectories and final responses and can work with agents outside LangChain. |
| Python tests and CI gates | DeepEval | End-to-end and component-level agent evaluation fit a test-oriented workflow. |
| Local matrix tests and red teaming | Promptfoo | The open-source CLI and library run repeatable evaluations and security cases in CI. |
| Product-specific policy or tool checks | Custom evaluators | Generic judges do not know your permissions, irreversible actions, budgets, or business invariants. |
Evaluate the steps and the outcome
A trajectory is the sequence of model decisions and tool calls in a run. A correct final answer can hide mistakes in those steps. An agent may call the wrong tool, retry needlessly, expose sensitive arguments, or arrive at a plausible answer without evidence. Conversely, a different but valid tool sequence should not fail merely because it differs from one golden trace.
Keep separate checks for:
- final-answer correctness and evidence support
- tool selection and argument validity
- required, forbidden, or repeated actions
- policy decisions and approval boundaries
- trajectory efficiency, latency, and cost
- recovery after tool errors or partial results
Use deterministic assertions where possible. Reserve model judges for semantic questions, calibrate them against reviewed examples, and store the judge model and prompt with every result.
For example, test an order-status agent with a user who cannot access the requested order. Check that it does not return the order’s details or call a write tool. A polite final refusal would still fail if an earlier step disclosed the private record.
A practical first stack
- In the application code that runs the model and tools, record a request ID, selected tool, redacted arguments, outcome, and duration.
- Turn production failures into a small regression dataset.
- Add deterministic checks for schemas, forbidden actions, budgets, and required evidence.
- Add one calibrated semantic judge for outcomes that rules cannot score.
- Run fast cases on each change and a broader suite before release.
- Sample production traces for new failure modes and promote them into tests.
Deeper reading
- Agent Evals: From Traces to Test Suites explains the evaluation design behind these tools.
- AI Agent Security covers policy checks that an evaluator should observe but cannot enforce by itself.
- Long-Running AI Agent Runtime covers the traces and checkpoints that make failures reproducible.