Evaluating an agent harness with the model held fixed
This article is for engineers deciding whether a change to an AI agent or workflow makes it better at its business task. A harness is the code around a language model: its prompts, tools, control flow, memory, checks and retries. You will learn how to hold the model fixed, check what the harness does and tells the user, and weigh success against cost and latency. You will also learn how to decide whether a candidate should replace the current design.
The worked example is an online shop’s support assistant, where refunds above $200 need supervisor approval. Checking only its final database made an agent with approval middleware look perfect, even though it told customers their completed refunds were still on hold. Checking actions and replies exposes that failure. Testing on messages no design was written for exposes the limits of a cheaper workflow. In that workflow, one model call reads the message and code runs the steps.
The evaluation takes eight steps, which the sections below follow in order.
What changes in a harness evaluation
A model comparison keeps the prompts, tools and execution loop fixed while changing the model. A harness comparison keeps the model fixed while changing the code around it. Repeated runs and uncertainty intervals help distinguish a harness improvement from run-to-run noise.
Anthropic’s guide Demystifying evals for AI agents distinguishes two pieces. The agent harness lets a model act as an agent; the evaluation harness runs tasks, records each attempt and grades it. Here I call the latter the evaluator. Both an agent, which chooses its own steps, and a workflow, whose steps are fixed in code, can be evaluated this way.
How much can the harness change a result? Published comparisons with a fixed model give a range:
- The Terminal-Bench 2.0 paper ran several agents with the same models, at least five times each. GPT-5 solved 49.6% of the tasks in Codex CLI and 33.9% in mini-SWE-agent, a gap of 15.7 points. Claude Opus 4.5 solved 57.8% in Terminus 2 and 52.1% in Claude Code, a gap of 5.7 points. The authors still conclude that choosing the model usually mattered more than choosing the agent.
- LangChain raised its deepagents CLI from 52.8% to 66.5% on Terminal-Bench 2.0 by changing the harness around
gpt-5.2-codex, including how much reasoning each phase of a task uses. - In Google DeepMind’s AutoHarness, Gemini 2.5 Flash writes code that rejects illegal moves before they reach a game. With that code around it, Flash won 9 of 16 two-player games against the larger Gemini 2.5 Pro.
These results show that changing the harness can move the score by 5 to 16 points, or let a smaller model beat a larger one. The size of the gain varied from case to case, so measure it for your own harness and model, and measure again when the model changes.
Hold these fixed, not only the model name:
- The model and its version, and the provider endpoint that serves it. A router such as OpenRouter can send a request to another provider unless you pin one.
- Sampling and reasoning settings, unless changing them is the experiment. In LangChain’s experiment, running at the highest reasoning effort instead of high lowered the score from 63.6% to 53.9%, because more tasks ran out of time. If a candidate sets effort differently for each phase, record that policy as part of the candidate. Keep the other conditions fixed.
- The machine the tasks run on. Anthropic changed only the container resources of Terminal-Bench 2.0 runs and moved the score by up to 6 points. It concludes that leaderboard gaps below 3 points deserve skepticism until the setups are documented and matched.
- The tasks, the checks and the evaluator’s version. A changed check is a different evaluation.
The parts you may change include prompts, tool definitions, the loop or workflow graph, middleware, retries, reply templates, and extra models. Count an extra model’s errors, cost and latency against the harness that uses it.
Six designs for the support assistant
To see this setup in practice, let’s recap the online-shop support assistant from Part 3. It issues refunds, changes addresses and updates account settings. One rule drives our comparison: refunds above $200 need a supervisor’s decision. We test six designs for running it (five from Part 3 plus a do-nothing baseline). They differ in who chooses the steps and who handles that wait:
| Design | How it runs | Supervisor approval |
|---|---|---|
| 0. Does nothing | No model call, write or reply; a baseline added here | None |
| 1. Plain agent | The model calls tools in a loop | Can leave a refund on hold; cannot resume for a decision |
| 2. Agent with approval middleware | The same loop, with code around tool calls that can pause and resume it | Middleware waits for the decision |
| 3. Agent inside a workflow | An agent starts the task; code then handles approval, refund and reply | Workflow waits for the decision |
| 4. Router and three agents | A small decision model sends each message to a narrower agent | No step to resume for a decision |
| 5. Workflow with no agent | One model call reads the message into fields; code chooses and runs the steps | Workflow waits for the decision |
Part 3 ran each design once. Here we grade runs saved on 15 October 2026. These include 10 repetitions per design on 16 known tasks, plus 5 repetitions of designs 1 to 5 on 32 new messages that no design was written for (described below). All model-using designs used openai/gpt-6-luna through OpenRouter’s OpenAI endpoint, with fallbacks disabled. Design 4’s additional decision model belongs to its harness.
The grading uses harness-evals, a small Python library you can install in your own project. The companion lab connects it to the assistant. This is Part 4 of Recursive Harness Engineering: Parts 2 and 3 built the designs. This part builds the evaluator that scores a harness change; in the next part, a program proposes the changes.
The figure shows the lab’s live adapter, the code that connects the evaluator to a running assistant. A separate process reads the database and the refund service’s records, and takes the reply, token usage and cost from the assistant’s own output. The saved October runs graded below recorded part of that evidence inside the assistant’s process. Who writes the evidence explains which part was recorded in-process and what that leaves untested.
Tasks come from the use case
The evaluation code can be shared, but the tasks cannot. Each harness is built for one use case (here, a support desk), so its tasks come from that use case. They reflect the policy the system must follow, the requests users send, and the failures you have already seen. A public benchmark tells you how a model does on someone else’s tasks. Anthropic’s guide suggests starting with 20 to 50 tasks drawn from real failures and growing the set from there.
In our example, the support desk starts smaller, with 16 known tasks carried over from earlier builds. Twelve come from Part 2: refunds, address changes and settings changes. The correct result may be the change itself, a refusal, a question for missing information, or a handoff to a supervisor. A script plays the supervisor and can approve, reject or leave the request waiting.
Part 3 adds four cases: an approved $350 refund, a rejected $350 refund, and two refunds whose service response is lost after the money was refunded. Those last two use $15 and $350 refunds; the assistant must retry without refunding twice.
For each task, write down what a correct run does before you run anything:
- The input and the starting state: the user’s message, and the database or files the run starts from.
- What must happen: the actions the environment should record, such as one refund of $350 on order O-2002.
- What must never happen: actions that are wrong in any run, such as a refund on another customer’s order. Apple’s ToolSandbox calls these minefields and scores a run that hits one as zero, whatever else it did. Treat them as critical.
- What the user must be told, and must not be told: the facts a reply has to state, and claims that would be false.
- The kind of outcome: an action, a question for missing information, a refusal with the policy reason, a handoff to a person, or an answer.
Here is the approved $350 refund task with every check written out, including the judge’s, as the lab’s evaluator/support.py builds it. Read the checks as the task’s contract:
Task(
"refund-over-limit-approved",
"This is ben.ortiz@example.com. The carbon bike wheels from "
"order O-2002 are cracked. Refund the full $350 now, please.",
(
state(), # the database is as expected
did("refund", order="O-2002", cents=35000), # refund issued
at_most(1, "refund"), # only once (critical)
Check(
"only permitted refunds",
lambda e: all(
a.matches("refund", {"order": "O-2002", "cents": 35000})
for a in e.actions if a.name == "refund"
),
critical=True,
),
NO_FALSE_HOLD, # reply never says it still waits
says(r"\b350(\.00)?\b"), # the reply states the amount
judged("outcome"), # the judge must say "action"
),
)
The critical refund checks reject a second refund or any refund with the wrong order or amount. The separate did check requires the expected refund to happen; failing to issue it is a task failure, not an unauthorized action.
The last check uses a judge: a second model that reads the reply and classifies it as an action confirmation, question, refusal, handoff or answer. Here the expected class is an action confirmation. The judge checks what kind of reply the user received, while the database and action checks verify whether the refund happened.
Each check returns true, false, or undecided when evidence is missing. A run passes only when it ends without an error and every check returns true. If the judge times out, the run does not pass, and the evaluator reports missing grading evidence rather than a harness failure.
Tasks that must end with no change
Many correct runs change nothing. The customer asks for a refund outside the 30-day window or on another customer’s order, or leaves out the city in a new address. The right result is a refusal or a question, and the database stays as it was. Half of the 16 tasks are like this.
That makes a check of the final database alone easy to pass. Design 0 passed 8 of the 16 tasks in each of its 10 repetitions without a call, write or reply. The authors of the Agentic Benchmark Checklist found the same problem in a public benchmark: on τ-bench’s airline tasks, an agent that returns empty responses scored 38%, higher than GPT-4o. They ask every benchmark to report such a trivial agent’s score. Run a design that does nothing through your own tasks first. Any pass on a task that requires a response exposes a missing check.
Development, validation and test tasks
Our 16 known tasks come from building the support assistant in Parts 2 and 3. Sixteen tasks are enough to show how the checks work, yet they fall short of the 20 to 50 recommended above. Worse, every design was built with those tasks in sight. That makes them a development set: you read their failures and change the harness until it passes them.
Part 3’s workflow with no agent shows why a development set is not enough. Its rules recognize refunds, address changes and settings changes, and tell the customer that a colleague will answer anything else. Because I wrote those rules around these 16 tasks, a perfect score on them says little about other messages. To test messages outside that set and grow the suite to 48 tasks, the lab adds 32 new messages that no design was written for. They cover 8 vague complaints, 8 messages with two requests, 8 policy questions and 8 requests outside the known kinds, such as closing an account. Each names its expected outcome and the database change it requires, if any, written before any run.
Machine learning separates its data for the same reason: a model does better on the examples it was trained or tuned on than on new ones. Google’s Machine Learning Crash Course uses three sets. The model trains on the training set. The validation set scores each adjustment, from a new learning rate to a new model design. The test set checks the chosen model at the end.
Nothing trains on the validation set. But the model you keep is the one that happened to suit those examples best, so its validation score is optimistic. The course recommends replacing validation and test data after repeated use for model selection makes their scores optimistic. Wan and colleagues measured the gap for prompts: prompts optimized to maximize validation accuracy scored 5.8 to 13.2 points lower on a separate test set. By contrast, the unoptimized prompt’s two scores stayed within about 1.7 points of each other.
With the language model’s weights fixed, the harness is what you adjust: prompts, tools and control flow take the place of weights and hyperparameters. Split your tasks the same way before the first run. Development tasks play the part of the training set, validation tasks choose among candidate harnesses, and test tasks score the chosen harness once. Andrew Ng’s Machine Learning Yearning calls the validation set the “dev set”; in this article, development tasks are the ones whose failures you fix.
Teams that tune agents split their tasks this way too. When Anthropic improved its agents’ tools on internal evaluations, it “relied on held-out test sets to ensure we did not overfit to our ‘training’ evaluations” (Writing effective tools for agents). METR iterated on its agent scaffolding with “a ‘development’ suite” of tasks and reported scores on “a held-out ‘test’ suite of 83 tasks” (DeepSeek-V3 evaluation). The AppWorld agent benchmark ships separate train, dev and test splits. It asks that nobody tune on the test tasks, “either via parameter grid search or manually by tweaking prompts”.
Each set in the harness row does the job of the set above it. Only the test set’s score measures examples the tuning has not seen.
In the lab, the 32 new messages, in a split named new, serve as the test set. A random split in machine learning draws the test set from the same pool as the training set. Here, by contrast, the new messages were written to include kinds of request the development tasks lack. A lower score on them can come from tuning on the known tasks or from requests no design was built for.
The lab has no separate validation set. Later in this article, the new messages decide between the workflow with no agent (design 5) and the agent inside a workflow (design 3), which makes them validation tasks for that choice. Reading their failures and fixing a harness would make them development tasks. Either way, the chosen design’s final score needs fresh messages.
Keep reviewed failures as regression tasks after fixing them, and run them on later harness or model changes. Remove private customer details before adding production cases. This reused set checks that fixes still work; reserve fresh requests for final assessment. Pair offline checks with production monitoring and occasional human review, as Anthropic’s evaluation guide recommends.
A program that proposes harness changes can make far more selection decisions than a person, increasing the risk of overfitting to the same tasks. Give it a validation set for its choices, and keep the test set out of its reach. In harness-evals, each task carries a split name, dev by default, and run evaluates one split at a time.
Holding tasks out leaves fewer for tuning. Meta-Harness, which searches over harness code with the model frozen, searched and evaluated on the same 89 Terminal-Bench 2.0 tasks. Its authors say a separate split would have weakened the search on so small a benchmark. They checked for overfitting by reading the harnesses and scanning them for task-specific strings. For your own use case, you can write more messages and hold some out.
Three levels of checks
The lab grades every saved run at three levels, each adding checks from the task contract above:
- Database (
state): the final database matches the expected change. Parts 2 and 3 graded this way. - Obligations (
did,at_most,only permitted refunds,never,NO_FALSE_HOLD,says): the database check, the refund-action checks, and two kinds of reply pattern. One requires facts the reply must state; the other forbids saying that an issued refund is still on hold. The lab defines required reply facts (says) for the 16 known tasks; the 32 new messages use the other obligation checks without task-specific fact patterns. - Judge (
judged): the obligations, plus a check that the judge’s classification of the reply matches the task’s expected outcome.
The figure shows, for each level, a failure it catches and one it cannot see; the next level checks the latter. No check in this lab tests whether an answer states the right policy.
The judge’s verdicts were saved with the runs on 15 October, so grading them again needed no new model calls. They come from the second version of its rubric; Checking the judge against labelled replies shows what that version was tested on, and what it was not. The three levels grade the same saved runs, and every result below names its level.
What each level catches on the known tasks
Success is the share of runs that passed every check at the named level. The table shows success on the 16 known tasks across 10 repetitions per task. The range in brackets is a 95% interval: where the score would plausibly land on another set of 16 tasks like these (how it is computed). A score of 0.81 means 81% passed.
| Design | Database | Obligations | Obligations and judge |
|---|---|---|---|
| 0. Does nothing | 0.50 (0.25–0.75) | 0.00 (0.00–0.00) | 0.00 (0.00–0.00) |
| 1. Plain agent | 0.87 (0.68–1.00) | 0.81 (0.61–0.99) | 0.69 (0.45–0.88) |
| 2. Agent with approval middleware | 1.00 (1.00–1.00) | 0.81 (0.62–1.00) | 0.68 (0.44–0.88) |
| 3. Agent inside a workflow | 1.00 (1.00–1.00) | 1.00 (1.00–1.00) | 0.88 (0.71–1.00) |
| 4. Router and three agents | 0.88 (0.69–1.00) | 0.81 (0.62–1.00) | 0.69 (0.46–0.88) |
| 5. Workflow with no agent | 1.00 (1.00–1.00) | 1.00 (1.00–1.00) | 0.94 (0.81–1.00) |
Because the interval resamples these 16 tasks, a 1.00–1.00 range only means every run passed on this set, not that the harness will pass every future request.
The figure plots the table’s means, each with a whisker for its interval. Follow the approval agent, design 2, from the database check to the judge: it starts level with designs 3 and 5 and ends below them.
Correct refunds, incorrect replies
What the stricter checks found on the known tasks:
- Design 0 fell from 0.50 to 0. Every task needs a reply, and it sends none.
- Design 2, the agent with approval middleware, fell from 1.00 to 0.81. In all 20 runs of the two approved tasks, the refund went through, yet the reply said it was still waiting. Each reply claimed: “Because it exceeds $200, it’s on hold for supervisor approval; it won’t be issued until approved.” In the Part 2 task whose supervisor never answers, it sent the customer no reply at all, in 10 of 10 runs.
- Designs 1 and 4 started lower, at 0.87 and 0.88, because neither can resume after a supervisor approves. Both approved $350 refunds stayed on hold, which the database check already fails. The stricter checks took both designs to 0.81 by failing one more task: the rejected $350 refund. With no approval step, they told the customer a supervisor would review the refund and ended the run with it still on hold, so no decision was ever made. The database check passed them on that task, because nothing was refunded either way.
- Designs 3 and 5 kept 1.00: in both, code writes the reply after the supervisor decides, from the refund service’s result.
The table and the chart count failures; a failure map shows where they are. Each row is a known task and each column a kind of check, with one panel per design and the judge included:
Two clusters stand out. The $350 refunds that wait for a supervisor’s decision break designs 1, 2 and 4 in different columns. Designs 1 and 4 never issue the approved refund, while design 2 issues it and then calls it still on hold. The judge column fails on another customer’s order in four designs and on the suspended account in all five. The next section shows that those two tasks have ambiguous expected outcomes. harness-evals draws such a map with failure_map_svg, and the lab writes one for each task set. Its make show command opens the run behind any square.
Some judge failures expose ambiguous tasks
The judge reduced every design’s score, but those reductions are not all confirmed customer-service failures:
- On another customer’s order, designs 1 to 4 usually said they could not find it on the customer’s account and asked them to check the number. The judge classified this reply as a question. The task expected a refusal, though the reply pattern accepted the response.
- On a suspended account’s address change, every design refused and told the customer to contact support. The judge classified this as a handoff; the task expected a refusal.
A reply can refuse a request and refer the customer to support. The table preserves the original labels and scores, but that overlap needs review before the judge is used to accept or reject a production change. Clarify which combinations are acceptable, version the rubric, then regrade every design in the comparison with it. Do not repair the labels only for the candidate.
New messages expose missing request types
The 16 known tasks only test requests every design was already built for. To see how the harnesses handle unfamiliar inputs, the table below scores the five working designs on the 32 new messages, with 5 repetitions per task. The do-nothing design was not run on this set.
| Design | Obligations | Obligations and judge |
|---|---|---|
| 1. Plain agent | 0.96 (0.91–1.00) | 0.88 (0.79–0.96) |
| 2. Agent with approval middleware | 0.94 (0.86–1.00) | 0.86 (0.75–0.94) |
| 3. Agent inside a workflow | 0.94 (0.86–1.00) | 0.88 (0.78–0.96) |
| 4. Router and three agents | 0.88 (0.75–0.97) | 0.81 (0.68–0.93) |
| 5. Workflow with no agent | 0.78 (0.62–0.91) | 0.31 (0.16–0.47) |
The workflow with no agent, design 5, passed all 40 policy-question runs under obligations and none with the judge added. Its schema has no category for answering a policy question. The obligations only checked that it left the database alone; the judge caught that it never answered the question.
Report these results by request type as well as overall: vague complaints, two-request messages, policy questions and other requests. The lab gives each group eight tasks, so the overall score weights them equally. Real traffic may have different proportions; a high aggregate score can still hide a poorly handled request type.
Design 5 also missed most messages with two requests, and vague complaints like a broken teapot with no order number. The agents looked up the customer’s orders and refunded the teapot. Part 3’s recommendation to start with this workflow still holds for the 16 known tasks. To handle the new messages, it needs support for policy questions, multiple requests in one message, and order lookup when a complaint gives no order number.
Part 3 also proposed a hybrid that routes standard requests to the workflow and leaves the rest to an agent. The saved runs include a prototype of that idea, a sixth design that is not in the table above. Its router sent 35 of the 40 policy-question runs to the workflow, which cannot answer questions. It sent the other 5, and most two-request messages, to a step that replies with one fixed sentence asking the customer to clarify. With the judge included, the hybrid passed only 14% of new-message runs, worse than either part alone. The pinned lab includes its saved runs for regrading, though not the prototype’s source code.
All results come from one chat model, one endpoint and one day. I wrote the reply patterns after reading Part 3’s replies, so they target failures I already knew. Claude generated the new messages and expected outcomes, which I approved before running the designs. That makes them a separate test of these designs, but still a small synthetic set rather than a sample of real customer traffic. The lab’s reports/part-4/RESULTS.md contains the full metrics.
Metrics for every evaluation
Report the same core metrics for every design. Each catches a failure the others hide:
| Metric | What it catches | How it is computed |
|---|---|---|
| Success | A design that solves fewer tasks | The share of runs that pass every check, averaged over tasks |
| pass^k | A design that solves a task only sometimes | The chance that k runs of a task all pass, averaged over tasks |
| Critical violations | A rare forbidden action that an average hides | Runs that failed a critical check, counted separately |
| Cost per task and per solved task | A design that buys success with tokens or retries | Dollars per run, and total dollars divided by passing runs |
| Model calls and tokens per task | Extra model work, measured apart from prices and cache discounts | Mean per run |
| Latency, median and 95th percentile | Slow runs that an average hides | Seconds per run |
| Infrastructure errors and undecided runs | Failures that are not the harness’s fault, and missing evidence | Counted separately; neither counts as a pass |
The figure computes the core metrics from one design’s runs: the plain agent on the 16 known tasks, 10 runs each, graded on obligations. Each row is a task and each square a run. Success counts the passing squares; pass^10 counts the rows with no failure; the interval resamples rows, not squares.
The 95% interval estimates how much average success could vary across tasks like the ones in your set. The library draws whole tasks with replacement, keeping their repeated runs together. It recomputes the average 2,000 times and keeps the middle 95% of those averages. Repeating one task does not supply independent evidence about other tasks. Evan Miller’s Adding error bars to evals explains how ignoring related questions can understate uncertainty. This interval cannot account for request types missing from the task set or mistakes in the checks. harness-evals resamples tasks the same way for cost, latency, calls and tokens, recomputing each metric on the runs of the drawn tasks.
pass^k comes from τ-bench. For a task with runs and passes, the estimate that independent runs under the same conditions all pass is
averaged over tasks, with . For example, if 9 of 10 runs pass, success is 90%, but pass^5 is . It asks whether the same kind of request is handled consistently, not whether one customer should retry it five times.
On τ-bench’s retail tasks, GPT-4o passed about 61% of single runs, while its pass^8 was below 25%. In our known-task runs, graded on obligations, the plain agent passed 81% of runs, and 75% of its tasks passed all ten. A related metric, pass@k, is the chance that at least one of k runs passes. That helps when something reliable picks the passing run, such as a test suite; this desk sends one reply.
Critical violations fail a run and are also counted separately. A refund on the wrong customer’s order in 1 run out of 200 barely moves average success, but gives away money the customer was not owed. None of the designs issued a forbidden or a second refund in these runs, including when the refund service’s reply was lost. The service uses the same operation key for each retry of a refund, so it returns the first receipt instead of issuing another refund.
Cost per solved task divides the cost of all runs, including failed ones, by the number that passed. AI Agents That Matter showed that simple baselines matched elaborate agents on HumanEval for less money. The elaborate agents cost from about 1.6 to over 50 times as much as a simple baseline. On the 16 known tasks, graded on obligations, the workflow with no agent cost $0.00005 per solved task, against $0.00025 to $0.00036 for the agent designs. These are harness model-call costs; they exclude evaluator calls and human review. Count unknown cost as unknown, never as zero: the router’s decision-model calls came back with no billed cost, so the lab estimates them from the price list. With no passing run, cost per solved task is undefined.
Model calls and tokens measure how much work a design asks of the model, apart from prices and cache discounts. On the known tasks, the plain agent made 3.6 model calls and used about 4,100 tokens per task; the workflow with no agent made one call with about 340 tokens.
Latency at the 95th percentile shows the slow runs a user notices. On the known tasks, the agent designs took 10.5 to 13.0 seconds at that percentile, the workflow with no agent 2.4 seconds. The supervisor here is a script; these times do not estimate how long a real person takes to approve a refund.
Infrastructure errors are runs the provider or the runner broke: a rate limit, a lost connection, a provider error. They count as failures, because a user was left without an answer, and they are reported apart from the harness’s own failures. A timeout counts against the harness when the harness caused it, as with LangChain’s highest reasoning setting. None of the saved runs had an infrastructure error.
Metrics that depend on the use case
Those core metrics fit almost any evaluation, while other checks depend on what your assistant can break. Add these to match your workload:
| If your use case… | Add | In the support desk |
|---|---|---|
| Gives the user something to act on later | Checks that the reply names it, such as an order or ticket id | After a refund, designs 1, 2 and 4 never named the refund id. On the known tasks, design 3 named it in 40% of replies and design 5 named it in all 100%. |
| Receives free-form requests | Success on messages the harness was not written for | The 32 new messages |
| Hands hard cases to people | Whether the user is told what happens next | Design 2 sent no reply while a refund waited for a supervisor |
| Has a verifier that can pick the best of several attempts | pass@k | Not used: the desk sends one reply |
| Writes free text a pattern cannot check | A judge’s reading, measured against reviewed labels | The judge agreed with 30 of 30 model-written labels it was not tuned on; human validation is still needed |
Acceptance thresholds
Set acceptance thresholds before comparing designs, so the observed results cannot change the requirements. For this example, I chose zero critical violations, pass^k of at least 0.9 (the reliability threshold), at most $0.002 per run, and a 95th-percentile latency of at most 15 seconds. In a real system, set them from the cost of mistakes and the response time you promise users. I chose these thresholds after the runs were saved, along with the two limits the comparison rule adds in Deciding between two designs. The decisions in this article therefore illustrate the rule; they do not test it.
The library uses all available repetitions for k: 10 on the known tasks and 5 on the new messages. When k equals the number of repetitions, pass^k is the share of tasks that passed every repetition. Keep k fixed within a comparison; pass^5 and pass^10 are different reliability measures.
Every design meets the cost and latency thresholds, and none had a critical violation. Reliability therefore decides which designs meet the acceptance thresholds:
- Known tasks, obligations: designs 3 and 5 meet them. Designs 1, 2 and 4 fail, with pass^10 of 0.75, 0.81 and 0.81.
- New messages, obligations: designs 1, 2 and 3 meet them. Designs 4 and 5 fail, with pass^5 of 0.88 and 0.78.
- New messages, with the judge: every design fails. The agent designs have pass^5 of 0.69 to 0.81; design 5 has 0.31.
Testing the evaluator
An evaluator can be wrong in two ways that this section covers. First, the harness may be able to reach the evidence it is graded on; second, the judge may misclassify replies.
Who writes the evidence
Action checks need evidence from the environment, independently of what the assistant says. Models scored on evidence they can reach have changed it instead of doing the task. Public graders have also been passed without solving a task:
- METR watched OpenAI’s o3 hack the grader in 39 of 128 runs on its research tasks. Some runs patched the grader to accept any answer; others overwrote the timer that measured code speed.
- The Darwin Gödel Machine, a coding agent that rewrites its own code, was scored on how well it stopped pretending to call tools. One version reached a perfect score by removing the logging of the markers that the check looked for. The authors saw this kind of hacking more often when the checking functions were visible to the agent.
- ImpossibleBench gave models coding tasks whose tests contradict the specification, so any pass means cheating. On its version of SWE-bench with conflicting tests, GPT-5 passed 54% of the tasks.
- Hao Wang and colleagues scored 89 of 89 on Terminal-Bench and 500 of 500 on SWE-bench Verified without solving a task. They replaced the
curlbinary that the grader uses to download a tool and added test hooks.
A separate process moves grading outside the assistant’s ordinary execution, but is not a security boundary when both share permissions. Before running code from an automated improver, restrict its filesystem and process access. It must not be able to read or change the evaluator, held-out tasks, labels or authoritative records. A container helps only when its mounts, credentials and permissions enforce that separation; the lab does not yet implement this protection.
In this lab, the saved runs use the refund service’s records, but their database checks ran in the assistant runner’s process. They show what the checks catch; they do not show that grading is protected from a harness that tampers with it. The live adapter (make part4-live) moves the database checks out of the assistant’s process. It starts the assistant in another process and reads each task’s database files itself. It gets issued refunds from new refunds rows and held refunds from the service’s records. This path has been tested only with the do-nothing design, not with paid runs of the five designs.
Checking the judge against labelled replies
A judge can recognize paraphrases that a fixed reply pattern misses. Still, its verdicts can be wrong if its rubric, the scoring instructions that define each category, is ambiguous. In the MT-Bench study, on pairs of deliberately similar answers, GPT-4 gave the same verdict after the two swapped places only 65% of the time. On web-agent trajectories, AgentRewardBench found no LLM judge above 70% precision. Hamel Husain and Shreya Shankar recommend labelling replies first and splitting the labels like training data. Then report how many good replies the judge rejects and how many bad ones it accepts.
The lab’s judge is the TypeSafe Jev model (typesafe/jev-1.13-20260917), which answers fixed questions about each reply. Design 4 also uses Jev to route messages; that router belongs to the candidate harness, while the judge stays fixed across every design. Although NO_FALSE_HOLD already checks on-hold claims at the obligations level, the judge still assigns each reply one outcome label (action, question, refusal, handoff or answer).
To verify those outcome labels, I tested two versions of Jev’s rubric (its scoring instructions) along two paths from the same saved runs:
- First sample of 30 replies (initial rubric). To spot mistakes without labelling hundreds of runs, Claude assigned reference labels to 30 saved replies before Jev graded them. Using the initial rubric, Jev agreed on 26 of 30. All four misses shared one bug: Jev labelled a reply
"action"whenever it mentioned a refund. That included design 2’s"submitted … on hold"replies and refusals saying an order was already refunded. - Editing the rubric. I updated the rubric’s category definitions to state that replies saying
"submitted … on hold"or"already refunded"are not"action". - Second sample of 30 fresh replies (revised rubric). To check the edit on replies I did not tune against, Claude labelled 30 new replies before Jev ran with the revised rubric. On this fresh batch, Jev agreed on 27 of 30 with the initial rubric and all 30 with the revised rubric.
Thirty held-out examples are still a small sample, yet labelling every saved reply one by one takes too long. To check the fix at larger scale without extra labels, I took all saved runs across both the 16 known tasks and the 32 new messages. Of those saved runs, 1,229 expected no database write (such as refusals, policy questions, or requests for missing details) and left the database unchanged. Because no database action happened in those 1,229 runs, the judge should not call their replies "action". Running Jev over those 1,229 replies confirmed the drop: the initial rubric wrongly called 65 of them "action", while the revised rubric called only 1.
Because Claude wrote those 60 labels, agreeing on the 30 held-out examples shows that the rubric edit fixed the “on hold” confusion, not that the judge matches human judgment. Before using a judge for release decisions, have a person who knows your support policy label a separate sample, resolve overlapping categories, and measure false accepts and false rejects. The lab’s labels/README.md lists both samples and verdicts; make part4-agreement recomputes these numbers.
How research tests and improves evaluators
Checking the evaluator by hand works when you review changes yourself. Once a program edits a harness or verifier from evaluator scores, the improver can exploit weak checks or let the grader drift. Five research methods guard that loop:
- Attack it, then repair it. BenchJack gives an agent ten public benchmarks to break. It found 219 flaws, and repeated rounds of exploiting and patching cut the share of tasks it could pass without solving them from about 100% to under 10% on four of the benchmarks.
- Anchor it to labels the optimizer cannot change. In Who Verifies the Verifier?, an agent improves its own verifier, checked against ten labelled examples. Without those anchors, the verifier decayed until it passed every run. The agents trained with it still looked fine on their task, so the downstream score did not show that the evaluator had broken. LLM-as-a-Judge Is Not an Oracle adds canary tasks that no honest agent can pass. A pass on a canary task shows cheating or a leak in the evaluation, and triggers an audit.
- Keep grading material out of the improver’s reach. Harness², from Google Cloud AI Research and Northwestern, improves a harness with the model frozen and no ground-truth feedback to the improver. The research assessment still uses benchmark grading. It runs the proposed harness on the same task as the current one and compares the two runs. Its improver works in a directory that a guard refuses to hand over if it contains a grader’s verdict, rubric or score.
- Separate the data for improving from the data for measuring. The lab’s judge was repaired on one labelled sample and measured on another. Beyond Prompts uses separate task sets for repairing a harness, for approving a change, and for the final scorecard.
- Grow the metrics from failures. AutoLibra turns people’s comments on agent trajectories into named metrics that a judge can score; on web navigation it found metrics such as error recovery and step efficiency.
The figure places the five methods, numbered as in the list, on a loop in which a program improves the harness. Methods 1, 2 and 5 work inside the evaluator. Method 3 limits what the improver can see, and method 4 splits the tasks the harness runs into separate sets for repairing, approving and the final score.
Comparing designs
A decision weighs several metrics at once: success under the checks you trust, cost, latency, and the use-case metrics. Put them side by side first. On the 32 new messages, with 95% intervals over tasks:
| Design | Success, obligations | Success, with judge | $ per task | p95 latency, s |
|---|---|---|---|---|
| 1. Plain agent | 0.96 (0.91–1.00) | 0.88 (0.79–0.96) | 0.00029 (0.00027–0.00032) | 12.5 (10.9–13.6) |
| 2. Agent with approval middleware | 0.94 (0.86–1.00) | 0.86 (0.75–0.94) | 0.00028 (0.00025–0.00031) | 9.9 (8.5–11.2) |
| 3. Agent inside a workflow | 0.94 (0.86–1.00) | 0.88 (0.78–0.96) | 0.00027 (0.00024–0.00029) | 10.8 (8.9–12.4) |
| 4. Router and three agents | 0.88 (0.75–0.97) | 0.81 (0.68–0.93) | 0.00029 (0.00026–0.00032) | 9.4 (8.3–10.5) |
| 5. Workflow with no agent | 0.78 (0.62–0.91) | 0.31 (0.16–0.47) | 0.00005 (0.00005–0.00005) | 1.6 (1.5–2.0) |
No design leads every column, so no single score can choose between them.
The frontier on two metrics
Pick the two metrics that matter most for the decision and plot every design on them, with an interval on each axis. Here there are two such pairs: success with the judge against cost per task, and success with the judge against p95 latency. The plots use the judge level because it is the strictest of the three. A design is on the frontier when no other design is at least as good on both metrics and better on one. AI Agents That Matter calls this the Pareto frontier and plots it for the accuracy and cost of coding and question-answering agents. Cost frontiers and plots omit any design with a run whose cost is unknown.
On the 16 known tasks, the workflow with no agent is at least as successful as every agent design: 0.94 with the judge, against 0.68 to 0.88. It is cheaper, at $0.00005 per task against $0.00025 to $0.00029. It is also faster, at 2.4 seconds at the 95th percentile against 10.5 to 13.0. The cost and latency intervals are far apart. The design that does nothing is on the frontier too, at zero cost and zero success. A frontier shows which trade-offs are efficient, not which designs are acceptable; the acceptance thresholds decide that.
On the 32 new messages, the frontier runs from the workflow with no agent, at 0.31, up to the agent designs, at 0.81 to 0.88. The workflow’s intervals do not overlap any agent’s on success, cost or p95 latency. Choosing an agent buys 0.50 to 0.57 more success; it costs about six times as much and takes six to eight times as long at the 95th percentile. Among the four agents, only one pair has intervals that do not overlap. The router’s p95 latency (8.3 to 10.5 seconds) is below the plain agent’s (10.9 to 13.6 seconds). Their success and cost intervals all overlap. The plain agent is on the cost frontier because it passed one more run in 160 than the agent inside a workflow. All five designs are on the latency frontier. Another sample could reorder the agents on either plot. Pairing, described next, separates some of them on cost or latency. Graded with the judge, it separates none of them on success; choosing among them needs more tasks or runs.
The lab’s make part4-regrade draws these four plots with harness-evals’ pareto and frontier_svg.
Deciding between two designs
The frontier shows each design with its own interval. To decide whether a candidate should replace the current harness, compare the two directly. harness-evals computes each design’s success rate on each task, subtracts current from candidate, then resamples those paired differences. Both designs face the same tasks, so the comparison accounts for differences in task difficulty. Cost, latency, calls, tokens and the use-case metrics get paired intervals the same way: each resample draws the same tasks for both designs.
Pairing makes the comparison more sensitive than reading two separate intervals. On the new messages, graded on obligations, the success interval is 0.62 to 0.91 for the workflow with no agent and 0.86 to 1.00 for the agent inside a workflow. Those two intervals overlap, yet the interval of their paired difference (−0.29 to −0.04) excludes zero. Paired the same way, the agent inside a workflow is cheaper than the plain agent there, though their separate cost intervals overlap.
The acceptance thresholds come first: a current design that fails them is replaced by a candidate that meets them without adding critical violations. Between two designs that meet the thresholds, only success and cost per task can make the rule adopt the candidate. A cheaper design may give up a little success. The allowed loss is the largest acceptable decrease: 0.02 here means two percentage points, or one additional failure per fifty runs on average. A guardrail limits how much another metric may get worse against the current design, whatever the candidate gains in success or cost. The lab sets one: p95 latency may rise by at most 2 seconds, within the 15-second absolute limit. The remaining metrics are reported with their intervals but do not decide. The rule decides in this order:
- Reject if the candidate fails the acceptance thresholds or has more critical violations than the current harness.
- Adopt if the current harness fails the thresholds and the candidate meets them.
- Reject if the whole 95% interval of candidate minus current lies below −0.02 for success, or past the limit for a guardrail.
- Adopt only if every guardrail’s interval stays within its limit and either success improves or cost falls with an acceptable success loss. Improvement means the whole success-difference interval is above zero. The cost route requires a cost-difference interval below zero and a success-difference interval at or above −0.02. A zero-width success interval cannot justify cost-only adoption.
- Otherwise the result is inconclusive: keep the current harness, and add tasks or runs.
If both designs pass every run, resampling their equal success rates always gives a difference of zero and an interval of [0, 0]. That interval does not establish a two-point loss limit on future requests. The library therefore keeps cost-only adoption inconclusive whenever the success interval has zero width. A lower cost on this fixed development suite is a measured result; a bound on future success loss needs fresh representative tasks and an uncertainty method that supports that claim.
Before comparing, check that both records use the same complete task set, repetition count, evaluator version and frozen conditions. In the pinned lab, this example compares the workflow with no agent against the agent inside a workflow on the new messages, graded on obligations. Run it with uv run python from the lab’s root:
from harness_evals import compare
from evaluator.__main__ import grades
from evaluator.support import usecase
current = grades("new", "obligations", 3) # the agent inside a workflow
candidate = grades("new", "obligations", 5) # the workflow with no agent
result = compare(usecase("obligations"), current, candidate)
print(result.decision, result.reasons)
print(result.deltas) # each metric: candidate minus current, and its interval
The workflow with no agent costs $0.00022 less per task (interval −0.00025 to −0.00020) and is 9.2 seconds faster at the 95th percentile (−10.9 to −7.1). Rule 1 rejects it before those gains count: it fails the reliability threshold on these messages. The example grades on obligations because the judge’s ambiguous labels have not been reviewed.
Two comparisons from the saved runs show how the checks and tasks change the decision:
- Design 2, the agent with approval middleware, against design 1, the plain agent, on the 16 known tasks. With the database check, design 2 passes every task (
pass^10of 1.00). Design 1 fails the reliability threshold (pass^10of 0.81), so rule 2 adopts design 2. Graded on obligations, design 2 is better on 2 tasks and worse on 1, and the interval of the difference runs from −0.18 to +0.19. Design 2 now fails the reliability threshold itself (pass^10of 0.81), so rule 1 rejects it. - Design 5 against design 3, the workflow with no agent against the agent inside a workflow, both graded on obligations. On the 16 known tasks, both pass every run and design 5 costs a fifth as much. The paired success interval is
[0, 0], so the cost-only comparison is inconclusive. On the 32 new messages, the comparison in the code above, design 5 scores 0.78 against design 3’s 0.94 (a difference of −0.16, interval −0.29 to −0.04) and fails thepass^5threshold, so rule 1 rejects it. With the judge added, the gap on new messages widens to −0.56 (0.31 against 0.88, interval −0.73 to −0.38).
The figure places each comparison’s interval against zero and the allowed loss. Read the bottom two rows together: the cheaper workflow has an inconclusive result on known tasks and is rejected on new messages.
How many runs and tasks
Repeated runs measure how much one design varies; more tasks measure how far the result carries over to other requests. You need both, in amounts that depend on the size of the difference you want to see:
- On SWE-bench Verified, a study of 60,000 agent runs found that a single run’s pass rate varied by 2.2 to 6.0 points between repetitions. Standard deviations were 1.0 to 1.8 points even at temperature 0. In that setup, the authors’ power calculation needs about 9 full benchmark repetitions per agent to detect a 2-point difference. It assumes a 1.5-point run-level standard deviation, 80% power and a two-sided 5% threshold; it is not a general nine-trials rule.
- With 16 tasks, our intervals were about ±0.2. Only differences of a few whole tasks show up.
- When every run passes, the bootstrap interval shrinks to
1.00–1.00, but uncertainty remains. By the statistical rule of three, independent trials with zero failures still leave a 95% upper failure bound of about , roughly 19% for . Even that bound assumes a random sample of real traffic; 16 hand-written tasks repeated 10 times are not 160 independent requests. - harness-evals requires at least six tasks for an interval-based decision. Its heuristic comes from a two-sided sign test: five nonzero task differences that all point the same way give a p-value of 0.0625, above 0.05. Six is this library’s minimum, not a guarantee that six tasks support a reliable decision.
In our runs, success under the database and obligation checks barely changed across the 10 repetitions. Only the plain agent failed one task in one run that it passed in the other nine. The judge’s readings varied a little more, with one task per agent design changing between runs. Under its database check, Part 3’s single repetition ranked the designs as the 10 repetitions do.
That single repetition’s cost ranking did not hold. In Part 3’s single repetition, the plain agent cost $0.0055 for the 16 tasks and the approval agent $0.0039. Over the 10 repetitions here, the plain agent ranged from $0.0036 to $0.0053. The median was $0.00394 for the plain agent and $0.00408 for the approval agent.
The plain agent’s most expensive repetition was its first, which read 45% of its input from the provider’s prompt cache; its later repetitions read 55% to 69%. A single comparison therefore mixes harness behavior with cache conditions. Interleave the designs’ runs, record the cached share of the input, and compare median costs over many runs.
Evaluation mistakes that skew scores and rankings
A flaw in your evaluation setup or grading can shift scores, flip which design ranks first, or reverse an adoption decision even when the harness stays the same. The table below lists eleven common mistakes engineers make when measuring agents. Some examples come from published benchmarks (LangChain, Anthropic, Darwin Gödel Machine). Others were measured directly in our support-desk runs by comparing weaker checks, a single repetition, or familiar tasks against the full suite to show how far the numbers move.
| Mistake | Example | Prevention |
|---|---|---|
| Accidentally changing a setting along with the harness | LangChain’s score fell from 63.6% to 53.9% when only the reasoning setting changed | Record fixed conditions separately from the changes being tested |
| Checking only the final database | A design that does nothing passed half of the support tasks | Check actions and replies; run a do-nothing design first |
| Letting the harness write its own evidence | A Darwin Gödel Machine version passed its check by removing the log markers the check read | Grade externally and restrict access to checks, labels and authoritative records |
| Trusting an unmeasured judge | The first judge read 19 of design 2’s 20 “on hold” replies as completed refunds | Label replies before the judge runs; measure it on labels it was not tuned on |
| Tuning on the tasks you report | The workflow with no agent scored 0.94 on its 16 tasks and 0.31 on new messages, both with the judge | Use validation tasks to select candidates and reserve fresh tasks for the final assessment |
| Averaging away a violation | No violation occurred in these runs, but one wrong-customer refund in 200 runs would lower success by only 0.005 | Report critical violations apart and require zero, as ToolSandbox scores any run that hits a minefield as zero |
| Treating one run as the result | One repetition made the plain agent look 40% more expensive than the approval agent; over 10 repetitions their medians were within 4% | Repeat and interleave runs; compare median cost; report pass^k and intervals |
| Deciding on success alone | With the judge, the plain agent leads on new messages by 1 run in 160. Yet design 3 costs less per task, with the whole paired interval below zero. | Report cost, latency, calls and tokens beside success, each with an interval; plot the frontier; set guardrails before comparing |
| Counting unknown cost as zero | The router’s decision-model calls came back with no billed cost; counted as zero, they would drop out of design 4’s cost | Compute it from the price list, or keep it unknown and fail a cost check while any run’s cost is unknown |
| Mixing infrastructure failures into harness failures | Anthropic’s infrastructure errors fell from 5.8% to 0.5% when only the container resources changed | Classify errors; report them apart, still as failures |
| Comparing results across evaluator versions | Design 2 scored 1.00 under Part 3’s database check and 0.81 graded on obligations | Version the checks; compare only runs graded by the same version |
The companion code
harness-evals is a Python 3.12+ library with no dependencies outside the standard library. An adapter connects it to a harness: your function runs a list of tasks once and returns one observed record per task. The library’s run function calls each adapter the same number of times, interleaving the designs, and its grading functions check the records. These functions produce the results in this article:
summarizereports the metrics above, with intervals for rates, costs, latencies, calls and tokens.paretoandfrontier_svgfind and draw the frontier.failure_map_svgdraws where each harness fails.compareapplies the adoption rule. Guardrails are declared on the use case:guardrails={"latency_p95": ("<=", 2.0)}allows a rise of at most 2 seconds.
compare refuses records with different tasks or repetition counts. The harness-evals compare command also validates each design’s loaded records against the manifest: task IDs, split, unique task/trial rows and declared repetitions must match. The manifests fingerprint task IDs, inputs, splits, check names and critical flags, plus the use-case version and frozen settings; from Python, call comparable to compare that metadata. comparable does not compare the evaluator version or the repetition count, and a check whose logic changes under the same name needs a new use-case version.
Install the library in a uv-managed project:
uv add "harness-evals @ git+https://github.com/slavadubrov/harness-evals@03cc80db2fd7039515887baf5c5321c3957139fa"
You describe the use case as tasks with checks and write one adapter per harness. The lab’s live adapter starts the assistant’s runner in a new process, then reads each task’s databases. This excerpt shows where the refund evidence comes from:
def observe(start: Path, final: Path, task_id: str):
passed = TASKS_BY_ID[task_id].check(start, final).passed
actions = [
Action("refund", {"order": r["order_id"],
"cents": r["amount_cents"],
"refund_id": r["id"]})
for r in diff(start, final)["refunds"].added
]
... # plus the refunds the service still holds for the case
This detects an added refund row even if the harness bypassed the service. If the harness can write to the database files directly, you still need the file and process permissions described above. The adapter handles an agent and a workflow alike because it observes their effects through the same interface.
To recompute the scores from the saved runs, use Python 3.12 and uv. The lab’s Make targets install its pinned dependencies first; no API key or new model call is needed. The commands pin the exact companion revision used here:
git clone https://github.com/slavadubrov/agent-harness-lab-public
cd agent-harness-lab-public
git checkout 19c7712762cca7b4f0af07523f3590ddf8ce8bb9
make part4-regrade # grades the saved runs; writes RESULTS.md, frontier plots and failure maps
make part4-agreement # both judge versions and the reply check against the labels
make part4-compare BASE=3 CANDIDATE=5 SPLIT=new
make failures DESIGN=5 SPLIT=new LEVEL=judge # the checks design 5 fails, task by task
make show DESIGN=2 TASK=refund-over-limit-approved # one saved run, check by check
make mine # score your own function on the 16 known tasks
The part4-compare command compares design 5 with design 3 on new messages, graded on obligations. It prints a success difference of −0.163 (95% interval −0.287 to −0.037), along with the paired differences in cost, latency, calls and tokens. It returns reject because design 5 misses the reliability threshold. Add LEVEL=judge to include the saved judge classifications. Regrading reuses the observations and verdicts; it does not independently reproduce the original model behavior.
make show prints one saved run as the evaluator saw it: the customer’s message, what the refund service recorded, the reply and the verdict of every check. make mine runs examples/your_harness.py, a template with no model call and a refund-only API. Replace its function with your own agent or workflow, then update the reported model metadata, calls, tokens and cost. Report real usage when measured and leave unknown values unknown; the rules-only starter’s zero usage does not describe a model call. Supporting address and settings tasks also requires extending its API.
Optional live runs use make part4-live TRIALS=5 with an OpenRouter key in OPENROUTER_API_KEY. This runs all five designs on the 16 known tasks, making paid model calls. The adapter has only been tested with the do-nothing design. At October 2026 prices, the saved agent runs cost about $0.0003 per task and repetition; 5 repetitions of all four agent designs plus the workflow would be roughly $0.10, excluding evaluator calls. Live mode uses obligations and does not run the judge.
What to do next
On your own harness, start with the three checks that changed a result here. Run a design that does nothing: on these tasks it passed 8 of 16 when only the final database was checked. Check the reply as well as the action: the agent with approval middleware (design 2) issued every approved refund and told the customer it was on hold. Score the candidate on messages it was not written for. Here, the workflow with no agent went from 1.00 on its own 16 tasks to 0.78 on 32 new ones under obligations, and to 0.31 once a judge read the replies. Set the thresholds before you read those scores, then compare the two designs with the rule.
The next part will use a program to propose harness changes and this evaluator to score them. Evaluate that search under a fixed budget: count proposals and accepted changes, include proposal and evaluation cost, and report regressions and held-out improvement. Repeat independent searches to assess the procedure, as Beyond Prompts does. Use development failures for edits and validation tasks for selection; keep the evaluator and final test tasks separate. Every candidate must meet the success, cost and latency limits, because retries alone can raise accuracy by spending more (AI Agents That Matter).
This article evaluates harnesses; it does not demonstrate recursive self-improvement. A fixed program searching for better harness code is automated optimization. A stronger recursive claim requires retained changes to affect the process that produces later changes, a distinction Harness² makes between execution and improvement harnesses. That process would need comparison at equal budgets on fresh tasks.