Recursive Harness Engineering · Part 4 of 4

Evaluating an agent harness with the model held fixed

This article is for engineers deciding whether a change to an AI agent or workflow makes it better at its business task. A harness is the code around a language model: its prompts, tools, control flow, memory, checks and retries. You will learn how to hold the model fixed, check what the harness does and tells the user, and weigh success against cost and latency. You will also learn how to decide whether a candidate should replace the current design.

The worked example is an online shop’s support assistant, where refunds above $200 need supervisor approval. Checking only its final database made an agent with approval middleware look perfect, even though it told customers their completed refunds were still on hold. Checking actions and replies exposes that failure. Testing on messages no design was written for exposes the limits of a cheaper workflow. In that workflow, one model call reads the message and code runs the steps.

The evaluation takes eight steps, which the sections below follow in order.

The eight steps of a harness evaluation in two rows of four: 1, fix the setup: model and settings; 2, write the tasks from the use case; 3, split the tasks and hold out a test set; 4, choose checks, from the database to a judge; 5, choose metrics, each with an interval; 6, set acceptance thresholds before comparing; 7, test the evaluator: its evidence and its judge; 8, compare designs: a frontier, then a rule.The eight steps of a harness evaluation in two rows of four: 1, fix the setup: model and settings; 2, write the tasks from the use case; 3, split the tasks and hold out a test set; 4, choose checks, from the database to a judge; 5, choose metrics, each with an interval; 6, set acceptance thresholds before comparing; 7, test the evaluator: its evidence and its judge; 8, compare designs: a frontier, then a rule.

What changes in a harness evaluation

A model comparison keeps the prompts, tools and execution loop fixed while changing the model. A harness comparison keeps the model fixed while changing the code around it. Repeated runs and uncertainty intervals help distinguish a harness improvement from run-to-run noise.

Anthropic’s guide Demystifying evals for AI agents distinguishes two pieces. The agent harness lets a model act as an agent; the evaluation harness runs tasks, records each attempt and grades it. Here I call the latter the evaluator. Both an agent, which chooses its own steps, and a workflow, whose steps are fixed in code, can be evaluated this way.

How much can the harness change a result? Published comparisons with a fixed model give a range:

  • The Terminal-Bench 2.0 paper ran several agents with the same models, at least five times each. GPT-5 solved 49.6% of the tasks in Codex CLI and 33.9% in mini-SWE-agent, a gap of 15.7 points. Claude Opus 4.5 solved 57.8% in Terminus 2 and 52.1% in Claude Code, a gap of 5.7 points. The authors still conclude that choosing the model usually mattered more than choosing the agent.
  • LangChain raised its deepagents CLI from 52.8% to 66.5% on Terminal-Bench 2.0 by changing the harness around gpt-5.2-codex, including how much reasoning each phase of a task uses.
  • In Google DeepMind’s AutoHarness, Gemini 2.5 Flash writes code that rejects illegal moves before they reach a game. With that code around it, Flash won 9 of 16 two-player games against the larger Gemini 2.5 Pro.

These results show that changing the harness can move the score by 5 to 16 points, or let a smaller model beat a larger one. The size of the gain varied from case to case, so measure it for your own harness and model, and measure again when the model changes.

Hold these fixed, not only the model name:

  • The model and its version, and the provider endpoint that serves it. A router such as OpenRouter can send a request to another provider unless you pin one.
  • Sampling and reasoning settings, unless changing them is the experiment. In LangChain’s experiment, running at the highest reasoning effort instead of high lowered the score from 63.6% to 53.9%, because more tasks ran out of time. If a candidate sets effort differently for each phase, record that policy as part of the candidate. Keep the other conditions fixed.
  • The machine the tasks run on. Anthropic changed only the container resources of Terminal-Bench 2.0 runs and moved the score by up to 6 points. It concludes that leaderboard gaps below 3 points deserve skepticism until the setups are documented and matched.
  • The tasks, the checks and the evaluator’s version. A changed check is a different evaluation.

The parts you may change include prompts, tool definitions, the loop or workflow graph, middleware, retries, reply templates, and extra models. Count an extra model’s errors, cost and latency against the harness that uses it.

Six designs for the support assistant

To see this setup in practice, let’s recap the online-shop support assistant from Part 3. It issues refunds, changes addresses and updates account settings. One rule drives our comparison: refunds above $200 need a supervisor’s decision. We test six designs for running it (five from Part 3 plus a do-nothing baseline). They differ in who chooses the steps and who handles that wait:

DesignHow it runsSupervisor approval
0. Does nothingNo model call, write or reply; a baseline added hereNone
1. Plain agentThe model calls tools in a loopCan leave a refund on hold; cannot resume for a decision
2. Agent with approval middlewareThe same loop, with code around tool calls that can pause and resume itMiddleware waits for the decision
3. Agent inside a workflowAn agent starts the task; code then handles approval, refund and replyWorkflow waits for the decision
4. Router and three agentsA small decision model sends each message to a narrower agentNo step to resume for a decision
5. Workflow with no agentOne model call reads the message into fields; code chooses and runs the stepsWorkflow waits for the decision

Part 3 ran each design once. Here we grade runs saved on 15 October 2026. These include 10 repetitions per design on 16 known tasks, plus 5 repetitions of designs 1 to 5 on 32 new messages that no design was written for (described below). All model-using designs used openai/gpt-6-luna through OpenRouter’s OpenAI endpoint, with fallbacks disabled. Design 4’s additional decision model belongs to its harness.

The grading uses harness-evals, a small Python library you can install in your own project. The companion lab connects it to the assistant. This is Part 4 of Recursive Harness Engineering: Parts 2 and 3 built the designs. This part builds the evaluator that scores a harness change; in the next part, a program proposes the changes.

The model, tasks, checks and environment stay fixed while the harness changes. A separate grading process reads the reply, usage and cost from the runner and the database records from the environment, grades each run and recommends a decision.The model, tasks, checks and environment stay fixed while the harness changes. A separate grading process reads the reply, usage and cost from the runner and the database records from the environment, grades each run and recommends a decision.

The figure shows the lab’s live adapter, the code that connects the evaluator to a running assistant. A separate process reads the database and the refund service’s records, and takes the reply, token usage and cost from the assistant’s own output. The saved October runs graded below recorded part of that evidence inside the assistant’s process. Who writes the evidence explains which part was recorded in-process and what that leaves untested.

Tasks come from the use case

The evaluation code can be shared, but the tasks cannot. Each harness is built for one use case (here, a support desk), so its tasks come from that use case. They reflect the policy the system must follow, the requests users send, and the failures you have already seen. A public benchmark tells you how a model does on someone else’s tasks. Anthropic’s guide suggests starting with 20 to 50 tasks drawn from real failures and growing the set from there.

In our example, the support desk starts smaller, with 16 known tasks carried over from earlier builds. Twelve come from Part 2: refunds, address changes and settings changes. The correct result may be the change itself, a refusal, a question for missing information, or a handoff to a supervisor. A script plays the supervisor and can approve, reject or leave the request waiting.

Part 3 adds four cases: an approved $350 refund, a rejected $350 refund, and two refunds whose service response is lost after the money was refunded. Those last two use $15 and $350 refunds; the assistant must retry without refunding twice.

For each task, write down what a correct run does before you run anything:

  1. The input and the starting state: the user’s message, and the database or files the run starts from.
  2. What must happen: the actions the environment should record, such as one refund of $350 on order O-2002.
  3. What must never happen: actions that are wrong in any run, such as a refund on another customer’s order. Apple’s ToolSandbox calls these minefields and scores a run that hits one as zero, whatever else it did. Treat them as critical.
  4. What the user must be told, and must not be told: the facts a reply has to state, and claims that would be false.
  5. The kind of outcome: an action, a question for missing information, a refusal with the policy reason, a handoff to a person, or an answer.

Here is the approved $350 refund task with every check written out, including the judge’s, as the lab’s evaluator/support.py builds it. Read the checks as the task’s contract:

Task(
    "refund-over-limit-approved",
    "This is ben.ortiz@example.com. The carbon bike wheels from "
    "order O-2002 are cracked. Refund the full $350 now, please.",
    (
        state(),                    # the database is as expected
        did("refund", order="O-2002", cents=35000),  # refund issued
        at_most(1, "refund"),       # only once (critical)
        Check(
            "only permitted refunds",
            lambda e: all(
                a.matches("refund", {"order": "O-2002", "cents": 35000})
                for a in e.actions if a.name == "refund"
            ),
            critical=True,
        ),
        NO_FALSE_HOLD,              # reply never says it still waits
        says(r"\b350(\.00)?\b"),    # the reply states the amount
        judged("outcome"),          # the judge must say "action"
    ),
)

The critical refund checks reject a second refund or any refund with the wrong order or amount. The separate did check requires the expected refund to happen; failing to issue it is a task failure, not an unauthorized action.

The last check uses a judge: a second model that reads the reply and classifies it as an action confirmation, question, refusal, handoff or answer. Here the expected class is an action confirmation. The judge checks what kind of reply the user received, while the database and action checks verify whether the refund happened.

Each check returns true, false, or undecided when evidence is missing. A run passes only when it ends without an error and every check returns true. If the judge times out, the run does not pass, and the evaluator reports missing grading evidence rather than a harness failure.

Tasks that must end with no change

Many correct runs change nothing. The customer asks for a refund outside the 30-day window or on another customer’s order, or leaves out the city in a new address. The right result is a refusal or a question, and the database stays as it was. Half of the 16 tasks are like this.

That makes a check of the final database alone easy to pass. Design 0 passed 8 of the 16 tasks in each of its 10 repetitions without a call, write or reply. The authors of the Agentic Benchmark Checklist found the same problem in a public benchmark: on τ-bench’s airline tasks, an agent that returns empty responses scored 38%, higher than GPT-4o. They ask every benchmark to report such a trivial agent’s score. Run a design that does nothing through your own tasks first. Any pass on a task that requires a response exposes a missing check.

Development, validation and test tasks

Our 16 known tasks come from building the support assistant in Parts 2 and 3. Sixteen tasks are enough to show how the checks work, yet they fall short of the 20 to 50 recommended above. Worse, every design was built with those tasks in sight. That makes them a development set: you read their failures and change the harness until it passes them.

Part 3’s workflow with no agent shows why a development set is not enough. Its rules recognize refunds, address changes and settings changes, and tell the customer that a colleague will answer anything else. Because I wrote those rules around these 16 tasks, a perfect score on them says little about other messages. To test messages outside that set and grow the suite to 48 tasks, the lab adds 32 new messages that no design was written for. They cover 8 vague complaints, 8 messages with two requests, 8 policy questions and 8 requests outside the known kinds, such as closing an account. Each names its expected outcome and the database change it requires, if any, written before any run.

Machine learning separates its data for the same reason: a model does better on the examples it was trained or tuned on than on new ones. Google’s Machine Learning Crash Course uses three sets. The model trains on the training set. The validation set scores each adjustment, from a new learning rate to a new model design. The test set checks the chosen model at the end.

Nothing trains on the validation set. But the model you keep is the one that happened to suit those examples best, so its validation score is optimistic. The course recommends replacing validation and test data after repeated use for model selection makes their scores optimistic. Wan and colleagues measured the gap for prompts: prompts optimized to maximize validation accuracy scored 5.8 to 13.2 points lower on a separate test set. By contrast, the unoptimized prompt’s two scores stayed within about 1.7 points of each other.

With the language model’s weights fixed, the harness is what you adjust: prompts, tools and control flow take the place of weights and hyperparameters. Split your tasks the same way before the first run. Development tasks play the part of the training set, validation tasks choose among candidate harnesses, and test tasks score the chosen harness once. Andrew Ng’s Machine Learning Yearning calls the validation set the “dev set”; in this article, development tasks are the ones whose failures you fix.

Teams that tune agents split their tasks this way too. When Anthropic improved its agents’ tools on internal evaluations, it “relied on held-out test sets to ensure we did not overfit to our ‘training’ evaluations” (Writing effective tools for agents). METR iterated on its agent scaffolding with “a ‘development’ suite” of tasks and reported scores on “a held-out ‘test’ suite of 83 tasks” (DeepSeek-V3 evaluation). The AppWorld agent benchmark ships separate train, dev and test splits. It asks that nobody tune on the test tasks, “either via parameter grid search or manually by tweaking prompts”.

In both rows, only the test set is kept out of tuning. In machine learning, a training set fits the weights and a validation set picks hyperparameters; both sit inside a dashed line, labelled optimistic scores, with the model they shape. A test set outside the line gives an unbiased score of the final model, used once. In harness tuning with the model held fixed, development tasks show what to fix and validation tasks pick a candidate; both sit inside a dashed line with the harness of prompts, tools and control flow. Test tasks outside the line give an unbiased score of the final harness, used once.In both rows, only the test set is kept out of tuning. In machine learning, a training set fits the weights and a validation set picks hyperparameters; both sit inside a dashed line, labelled optimistic scores, with the model they shape. A test set outside the line gives an unbiased score of the final model, used once. In harness tuning with the model held fixed, development tasks show what to fix and validation tasks pick a candidate; both sit inside a dashed line with the harness of prompts, tools and control flow. Test tasks outside the line give an unbiased score of the final harness, used once.

Each set in the harness row does the job of the set above it. Only the test set’s score measures examples the tuning has not seen.

In the lab, the 32 new messages, in a split named new, serve as the test set. A random split in machine learning draws the test set from the same pool as the training set. Here, by contrast, the new messages were written to include kinds of request the development tasks lack. A lower score on them can come from tuning on the known tasks or from requests no design was built for.

The lab has no separate validation set. Later in this article, the new messages decide between the workflow with no agent (design 5) and the agent inside a workflow (design 3), which makes them validation tasks for that choice. Reading their failures and fixing a harness would make them development tasks. Either way, the chosen design’s final score needs fresh messages.

Keep reviewed failures as regression tasks after fixing them, and run them on later harness or model changes. Remove private customer details before adding production cases. This reused set checks that fixes still work; reserve fresh requests for final assessment. Pair offline checks with production monitoring and occasional human review, as Anthropic’s evaluation guide recommends.

A program that proposes harness changes can make far more selection decisions than a person, increasing the risk of overfitting to the same tasks. Give it a validation set for its choices, and keep the test set out of its reach. In harness-evals, each task carries a split name, dev by default, and run evaluates one split at a time.

Holding tasks out leaves fewer for tuning. Meta-Harness, which searches over harness code with the model frozen, searched and evaluated on the same 89 Terminal-Bench 2.0 tasks. Its authors say a separate split would have weakened the search on so small a benchmark. They checked for overfitting by reading the harnesses and scanning them for task-specific strings. For your own use case, you can write more messages and hold some out.

Three levels of checks

The lab grades every saved run at three levels, each adding checks from the task contract above:

  • Database (state): the final database matches the expected change. Parts 2 and 3 graded this way.
  • Obligations (did, at_most, only permitted refunds, never, NO_FALSE_HOLD, says): the database check, the refund-action checks, and two kinds of reply pattern. One requires facts the reply must state; the other forbids saying that an issued refund is still on hold. The lab defines required reply facts (says) for the 16 known tasks; the 32 new messages use the other obligation checks without task-specific fact patterns.
  • Judge (judged): the obligations, plus a check that the judge’s classification of the reply matches the task’s expected outcome.

Three columns, one per level of checks. Database reads the final database with the state check; it catches a refund that was never issued, and misses what the reply says, such as design 2 calling its issued refunds still on hold. Obligations keep the state check and add did, at_most, only permitted refunds, never, NO_FALSE_HOLD and says, reading the refund service's records and the reply; they catch design 2 calling an issued $350 refund still on hold, and miss whether a reply answers: design 5 passes 40 of 40 policy-question runs. Judge keeps all of those and adds judged; it catches design 5 leaving policy questions unanswered, and misses whether the answer states the right policy. Arrows lead from each level's misses to the next level's catches.Three columns, one per level of checks. Database reads the final database with the state check; it catches a refund that was never issued, and misses what the reply says, such as design 2 calling its issued refunds still on hold. Obligations keep the state check and add did, at_most, only permitted refunds, never, NO_FALSE_HOLD and says, reading the refund service's records and the reply; they catch design 2 calling an issued $350 refund still on hold, and miss whether a reply answers: design 5 passes 40 of 40 policy-question runs. Judge keeps all of those and adds judged; it catches design 5 leaving policy questions unanswered, and misses whether the answer states the right policy. Arrows lead from each level's misses to the next level's catches.

The figure shows, for each level, a failure it catches and one it cannot see; the next level checks the latter. No check in this lab tests whether an answer states the right policy.

The judge’s verdicts were saved with the runs on 15 October, so grading them again needed no new model calls. They come from the second version of its rubric; Checking the judge against labelled replies shows what that version was tested on, and what it was not. The three levels grade the same saved runs, and every result below names its level.

What each level catches on the known tasks

Success is the share of runs that passed every check at the named level. The table shows success on the 16 known tasks across 10 repetitions per task. The range in brackets is a 95% interval: where the score would plausibly land on another set of 16 tasks like these (how it is computed). A score of 0.81 means 81% passed.

DesignDatabaseObligationsObligations and judge
0. Does nothing0.50 (0.25–0.75)0.00 (0.00–0.00)0.00 (0.00–0.00)
1. Plain agent0.87 (0.68–1.00)0.81 (0.61–0.99)0.69 (0.45–0.88)
2. Agent with approval middleware1.00 (1.00–1.00)0.81 (0.62–1.00)0.68 (0.44–0.88)
3. Agent inside a workflow1.00 (1.00–1.00)1.00 (1.00–1.00)0.88 (0.71–1.00)
4. Router and three agents0.88 (0.69–1.00)0.81 (0.62–1.00)0.69 (0.46–0.88)
5. Workflow with no agent1.00 (1.00–1.00)1.00 (1.00–1.00)0.94 (0.81–1.00)

Because the interval resamples these 16 tasks, a 1.00–1.00 range only means every run passed on this set, not that the harness will pass every future request.

Bars compare the known-task success rates listed in the preceding table. The approval agent is highlighted: its score falls from 1.00 on database checks to 0.81 on obligations and 0.68 with the judge. Each bar has a whisker for its 95% interval over tasks.Bars compare the known-task success rates listed in the preceding table. The approval agent is highlighted: its score falls from 1.00 on database checks to 0.81 on obligations and 0.68 with the judge. Each bar has a whisker for its 95% interval over tasks.

The figure plots the table’s means, each with a whisker for its interval. Follow the approval agent, design 2, from the database check to the judge: it starts level with designs 3 and 5 and ends below them.

Correct refunds, incorrect replies

What the stricter checks found on the known tasks:

  • Design 0 fell from 0.50 to 0. Every task needs a reply, and it sends none.
  • Design 2, the agent with approval middleware, fell from 1.00 to 0.81. In all 20 runs of the two approved tasks, the refund went through, yet the reply said it was still waiting. Each reply claimed: “Because it exceeds $200, it’s on hold for supervisor approval; it won’t be issued until approved.” In the Part 2 task whose supervisor never answers, it sent the customer no reply at all, in 10 of 10 runs.
  • Designs 1 and 4 started lower, at 0.87 and 0.88, because neither can resume after a supervisor approves. Both approved $350 refunds stayed on hold, which the database check already fails. The stricter checks took both designs to 0.81 by failing one more task: the rejected $350 refund. With no approval step, they told the customer a supervisor would review the refund and ended the run with it still on hold, so no decision was ever made. The database check passed them on that task, because nothing was refunded either way.
  • Designs 3 and 5 kept 1.00: in both, code writes the reply after the supervisor decides, from the refund service’s result.

The table and the chart count failures; a failure map shows where they are. Each row is a known task and each column a kind of check, with one panel per design and the judge included:

A grid of the 16 known tasks in rows, grouped by kind of request, and eight kinds of check in columns, with one panel for each of designs 1 to 5. Each square shows how many of 10 runs, graded with the judge, failed that check. The permitted-refund check has no failures. Failures concentrate in the $350 refund rows for designs 1, 2 and 4, and in the judge column on two other tasks: another customer's order and the suspended account.A grid of the 16 known tasks in rows, grouped by kind of request, and eight kinds of check in columns, with one panel for each of designs 1 to 5. Each square shows how many of 10 runs, graded with the judge, failed that check. The permitted-refund check has no failures. Failures concentrate in the $350 refund rows for designs 1, 2 and 4, and in the judge column on two other tasks: another customer's order and the suspended account.

Two clusters stand out. The $350 refunds that wait for a supervisor’s decision break designs 1, 2 and 4 in different columns. Designs 1 and 4 never issue the approved refund, while design 2 issues it and then calls it still on hold. The judge column fails on another customer’s order in four designs and on the suspended account in all five. The next section shows that those two tasks have ambiguous expected outcomes. harness-evals draws such a map with failure_map_svg, and the lab writes one for each task set. Its make show command opens the run behind any square.

Some judge failures expose ambiguous tasks

The judge reduced every design’s score, but those reductions are not all confirmed customer-service failures:

  • On another customer’s order, designs 1 to 4 usually said they could not find it on the customer’s account and asked them to check the number. The judge classified this reply as a question. The task expected a refusal, though the reply pattern accepted the response.
  • On a suspended account’s address change, every design refused and told the customer to contact support. The judge classified this as a handoff; the task expected a refusal.

A reply can refuse a request and refer the customer to support. The table preserves the original labels and scores, but that overlap needs review before the judge is used to accept or reject a production change. Clarify which combinations are acceptable, version the rubric, then regrade every design in the comparison with it. Do not repair the labels only for the candidate.

New messages expose missing request types

The 16 known tasks only test requests every design was already built for. To see how the harnesses handle unfamiliar inputs, the table below scores the five working designs on the 32 new messages, with 5 repetitions per task. The do-nothing design was not run on this set.

DesignObligationsObligations and judge
1. Plain agent0.96 (0.91–1.00)0.88 (0.79–0.96)
2. Agent with approval middleware0.94 (0.86–1.00)0.86 (0.75–0.94)
3. Agent inside a workflow0.94 (0.86–1.00)0.88 (0.78–0.96)
4. Router and three agents0.88 (0.75–0.97)0.81 (0.68–0.93)
5. Workflow with no agent0.78 (0.62–0.91)0.31 (0.16–0.47)

The workflow with no agent, design 5, passed all 40 policy-question runs under obligations and none with the judge added. Its schema has no category for answering a policy question. The obligations only checked that it left the database alone; the judge caught that it never answered the question.

Report these results by request type as well as overall: vague complaints, two-request messages, policy questions and other requests. The lab gives each group eight tasks, so the overall score weights them equally. Real traffic may have different proportions; a high aggregate score can still hide a poorly handled request type.

Design 5 also missed most messages with two requests, and vague complaints like a broken teapot with no order number. The agents looked up the customer’s orders and refunded the teapot. Part 3’s recommendation to start with this workflow still holds for the 16 known tasks. To handle the new messages, it needs support for policy questions, multiple requests in one message, and order lookup when a complaint gives no order number.

Part 3 also proposed a hybrid that routes standard requests to the workflow and leaves the rest to an agent. The saved runs include a prototype of that idea, a sixth design that is not in the table above. Its router sent 35 of the 40 policy-question runs to the workflow, which cannot answer questions. It sent the other 5, and most two-request messages, to a step that replies with one fixed sentence asking the customer to clarify. With the judge included, the hybrid passed only 14% of new-message runs, worse than either part alone. The pinned lab includes its saved runs for regrading, though not the prototype’s source code.

All results come from one chat model, one endpoint and one day. I wrote the reply patterns after reading Part 3’s replies, so they target failures I already knew. Claude generated the new messages and expected outcomes, which I approved before running the designs. That makes them a separate test of these designs, but still a small synthetic set rather than a sample of real customer traffic. The lab’s reports/part-4/RESULTS.md contains the full metrics.

Metrics for every evaluation

Report the same core metrics for every design. Each catches a failure the others hide:

MetricWhat it catchesHow it is computed
SuccessA design that solves fewer tasksThe share of runs that pass every check, averaged over tasks
pass^kA design that solves a task only sometimesThe chance that k runs of a task all pass, averaged over tasks
Critical violationsA rare forbidden action that an average hidesRuns that failed a critical check, counted separately
Cost per task and per solved taskA design that buys success with tokens or retriesDollars per run, and total dollars divided by passing runs
Model calls and tokens per taskExtra model work, measured apart from prices and cache discountsMean per run
Latency, median and 95th percentileSlow runs that an average hidesSeconds per run
Infrastructure errors and undecided runsFailures that are not the harness’s fault, and missing evidenceCounted separately; neither counts as a pass

The plain agent's 160 runs on the 16 known tasks, graded on obligations, as a grid of 16 task rows by 10 runs. Twelve rows pass every run, so pass^10 is 12 of 16, or 0.75. One row passes 9 of 10 runs, which counts toward success but not toward pass^10. Three rows fail every run: the $350 refunds on which a supervisor decides, approved, rejected, and approved with the refund service's response lost. Success is 129 of 160 runs, 0.81, with a 95% interval of 0.61 to 0.99 from drawing 16 whole rows with replacement 2,000 times. Below, the 160 run latencies lie on an axis from 0 to 20 seconds, with the median at 7.3 seconds and the 95th percentile at 11.6 seconds marked. Cost is $0.00026 per run and $0.00032 per solved task; there were no critical violations.The plain agent's 160 runs on the 16 known tasks, graded on obligations, as a grid of 16 task rows by 10 runs. Twelve rows pass every run, so pass^10 is 12 of 16, or 0.75. One row passes 9 of 10 runs, which counts toward success but not toward pass^10. Three rows fail every run: the $350 refunds on which a supervisor decides, approved, rejected, and approved with the refund service's response lost. Success is 129 of 160 runs, 0.81, with a 95% interval of 0.61 to 0.99 from drawing 16 whole rows with replacement 2,000 times. Below, the 160 run latencies lie on an axis from 0 to 20 seconds, with the median at 7.3 seconds and the 95th percentile at 11.6 seconds marked. Cost is $0.00026 per run and $0.00032 per solved task; there were no critical violations.

The figure computes the core metrics from one design’s runs: the plain agent on the 16 known tasks, 10 runs each, graded on obligations. Each row is a task and each square a run. Success counts the passing squares; pass^10 counts the rows with no failure; the interval resamples rows, not squares.

The 95% interval estimates how much average success could vary across tasks like the ones in your set. The library draws whole tasks with replacement, keeping their repeated runs together. It recomputes the average 2,000 times and keeps the middle 95% of those averages. Repeating one task does not supply independent evidence about other tasks. Evan Miller’s Adding error bars to evals explains how ignoring related questions can understate uncertainty. This interval cannot account for request types missing from the task set or mistakes in the checks. harness-evals resamples tasks the same way for cost, latency, calls and tokens, recomputing each metric on the runs of the drawn tasks.

pass^k comes from τ-bench. For a task with nn runs and cc passes, the estimate that kk independent runs under the same conditions all pass is

passk=(ck)/(nk),\text{pass}^k = \binom{c}{k} \Big/ \binom{n}{k},

averaged over tasks, with 1≤k≤n1 \leq k \leq n. For example, if 9 of 10 runs pass, success is 90%, but pass^5 is (95)/(105)=50%\binom{9}{5}/\binom{10}{5}=50\%. It asks whether the same kind of request is handled consistently, not whether one customer should retry it five times.

On τ-bench’s retail tasks, GPT-4o passed about 61% of single runs, while its pass^8 was below 25%. In our known-task runs, graded on obligations, the plain agent passed 81% of runs, and 75% of its tasks passed all ten. A related metric, pass@k, is the chance that at least one of k runs passes. That helps when something reliable picks the passing run, such as a test suite; this desk sends one reply.

Critical violations fail a run and are also counted separately. A refund on the wrong customer’s order in 1 run out of 200 barely moves average success, but gives away money the customer was not owed. None of the designs issued a forbidden or a second refund in these runs, including when the refund service’s reply was lost. The service uses the same operation key for each retry of a refund, so it returns the first receipt instead of issuing another refund.

Cost per solved task divides the cost of all runs, including failed ones, by the number that passed. AI Agents That Matter showed that simple baselines matched elaborate agents on HumanEval for less money. The elaborate agents cost from about 1.6 to over 50 times as much as a simple baseline. On the 16 known tasks, graded on obligations, the workflow with no agent cost $0.00005 per solved task, against $0.00025 to $0.00036 for the agent designs. These are harness model-call costs; they exclude evaluator calls and human review. Count unknown cost as unknown, never as zero: the router’s decision-model calls came back with no billed cost, so the lab estimates them from the price list. With no passing run, cost per solved task is undefined.

Model calls and tokens measure how much work a design asks of the model, apart from prices and cache discounts. On the known tasks, the plain agent made 3.6 model calls and used about 4,100 tokens per task; the workflow with no agent made one call with about 340 tokens.

Latency at the 95th percentile shows the slow runs a user notices. On the known tasks, the agent designs took 10.5 to 13.0 seconds at that percentile, the workflow with no agent 2.4 seconds. The supervisor here is a script; these times do not estimate how long a real person takes to approve a refund.

Infrastructure errors are runs the provider or the runner broke: a rate limit, a lost connection, a provider error. They count as failures, because a user was left without an answer, and they are reported apart from the harness’s own failures. A timeout counts against the harness when the harness caused it, as with LangChain’s highest reasoning setting. None of the saved runs had an infrastructure error.

Metrics that depend on the use case

Those core metrics fit almost any evaluation, while other checks depend on what your assistant can break. Add these to match your workload:

If your use case…AddIn the support desk
Gives the user something to act on laterChecks that the reply names it, such as an order or ticket idAfter a refund, designs 1, 2 and 4 never named the refund id. On the known tasks, design 3 named it in 40% of replies and design 5 named it in all 100%.
Receives free-form requestsSuccess on messages the harness was not written forThe 32 new messages
Hands hard cases to peopleWhether the user is told what happens nextDesign 2 sent no reply while a refund waited for a supervisor
Has a verifier that can pick the best of several attemptspass@kNot used: the desk sends one reply
Writes free text a pattern cannot checkA judge’s reading, measured against reviewed labelsThe judge agreed with 30 of 30 model-written labels it was not tuned on; human validation is still needed

Acceptance thresholds

Set acceptance thresholds before comparing designs, so the observed results cannot change the requirements. For this example, I chose zero critical violations, pass^k of at least 0.9 (the reliability threshold), at most $0.002 per run, and a 95th-percentile latency of at most 15 seconds. In a real system, set them from the cost of mistakes and the response time you promise users. I chose these thresholds after the runs were saved, along with the two limits the comparison rule adds in Deciding between two designs. The decisions in this article therefore illustrate the rule; they do not test it.

The library uses all available repetitions for k: 10 on the known tasks and 5 on the new messages. When k equals the number of repetitions, pass^k is the share of tasks that passed every repetition. Keep k fixed within a comparison; pass^5 and pass^10 are different reliability measures.

Every design meets the cost and latency thresholds, and none had a critical violation. Reliability therefore decides which designs meet the acceptance thresholds:

  • Known tasks, obligations: designs 3 and 5 meet them. Designs 1, 2 and 4 fail, with pass^10 of 0.75, 0.81 and 0.81.
  • New messages, obligations: designs 1, 2 and 3 meet them. Designs 4 and 5 fail, with pass^5 of 0.88 and 0.78.
  • New messages, with the judge: every design fails. The agent designs have pass^5 of 0.69 to 0.81; design 5 has 0.31.

Testing the evaluator

An evaluator can be wrong in two ways that this section covers. First, the harness may be able to reach the evidence it is graded on; second, the judge may misclassify replies.

Who writes the evidence

Action checks need evidence from the environment, independently of what the assistant says. Models scored on evidence they can reach have changed it instead of doing the task. Public graders have also been passed without solving a task:

  • METR watched OpenAI’s o3 hack the grader in 39 of 128 runs on its research tasks. Some runs patched the grader to accept any answer; others overwrote the timer that measured code speed.
  • The Darwin Gödel Machine, a coding agent that rewrites its own code, was scored on how well it stopped pretending to call tools. One version reached a perfect score by removing the logging of the markers that the check looked for. The authors saw this kind of hacking more often when the checking functions were visible to the agent.
  • ImpossibleBench gave models coding tasks whose tests contradict the specification, so any pass means cheating. On its version of SWE-bench with conflicting tests, GPT-5 passed 54% of the tasks.
  • Hao Wang and colleagues scored 89 of 89 on Terminal-Bench and 500 of 500 on SWE-bench Verified without solving a task. They replaced the curl binary that the grader uses to download a tool and added test hooks.

A separate process moves grading outside the assistant’s ordinary execution, but is not a security boundary when both share permissions. Before running code from an automated improver, restrict its filesystem and process access. It must not be able to read or change the evaluator, held-out tasks, labels or authoritative records. A container helps only when its mounts, credentials and permissions enforce that separation; the lab does not yet implement this protection.

In this lab, the saved runs use the refund service’s records, but their database checks ran in the assistant runner’s process. They show what the checks catch; they do not show that grading is protected from a harness that tampers with it. The live adapter (make part4-live) moves the database checks out of the assistant’s process. It starts the assistant in another process and reads each task’s database files itself. It gets issued refunds from new refunds rows and held refunds from the service’s records. This path has been tested only with the do-nothing design, not with paid runs of the five designs.

Checking the judge against labelled replies

A judge can recognize paraphrases that a fixed reply pattern misses. Still, its verdicts can be wrong if its rubric, the scoring instructions that define each category, is ambiguous. In the MT-Bench study, on pairs of deliberately similar answers, GPT-4 gave the same verdict after the two swapped places only 65% of the time. On web-agent trajectories, AgentRewardBench found no LLM judge above 70% precision. Hamel Husain and Shreya Shankar recommend labelling replies first and splitting the labels like training data. Then report how many good replies the judge rejects and how many bad ones it accepts.

The lab’s judge is the TypeSafe Jev model (typesafe/jev-1.13-20260917), which answers fixed questions about each reply. Design 4 also uses Jev to route messages; that router belongs to the candidate harness, while the judge stays fixed across every design. Although NO_FALSE_HOLD already checks on-hold claims at the obligations level, the judge still assigns each reply one outcome label (action, question, refusal, handoff or answer).

To verify those outcome labels, I tested two versions of Jev’s rubric (its scoring instructions) along two paths from the same saved runs:

Two parallel checks from the saved support-desk replies across the 16 known tasks and 32 new messages. On the left, labelled samples of 60 replies: 1, Claude labels 30 saved replies before Jev grades them with the initial rubric, matching 26 of 30 with 4 misses calling 'submitted ... on hold' or 'already refunded' an action; 2, Jev's rubric is edited to exclude those phrases from action; 3, on 30 fresh replies labelled by Claude before re-running, Jev matches 27 of 30 with the initial rubric and all 30 with the revised rubric. On the right, a no-label filter keeps all runs where the task expected no database write and the database stayed untouched, giving 1,229 no-action runs; Jev's false action calls on those 1,229 drop from 65 with the initial rubric to 1 with the revised rubric.Two parallel checks from the saved support-desk replies across the 16 known tasks and 32 new messages. On the left, labelled samples of 60 replies: 1, Claude labels 30 saved replies before Jev grades them with the initial rubric, matching 26 of 30 with 4 misses calling 'submitted ... on hold' or 'already refunded' an action; 2, Jev's rubric is edited to exclude those phrases from action; 3, on 30 fresh replies labelled by Claude before re-running, Jev matches 27 of 30 with the initial rubric and all 30 with the revised rubric. On the right, a no-label filter keeps all runs where the task expected no database write and the database stayed untouched, giving 1,229 no-action runs; Jev's false action calls on those 1,229 drop from 65 with the initial rubric to 1 with the revised rubric.

  1. First sample of 30 replies (initial rubric). To spot mistakes without labelling hundreds of runs, Claude assigned reference labels to 30 saved replies before Jev graded them. Using the initial rubric, Jev agreed on 26 of 30. All four misses shared one bug: Jev labelled a reply "action" whenever it mentioned a refund. That included design 2’s "submitted … on hold" replies and refusals saying an order was already refunded.
  2. Editing the rubric. I updated the rubric’s category definitions to state that replies saying "submitted … on hold" or "already refunded" are not "action".
  3. Second sample of 30 fresh replies (revised rubric). To check the edit on replies I did not tune against, Claude labelled 30 new replies before Jev ran with the revised rubric. On this fresh batch, Jev agreed on 27 of 30 with the initial rubric and all 30 with the revised rubric.

Thirty held-out examples are still a small sample, yet labelling every saved reply one by one takes too long. To check the fix at larger scale without extra labels, I took all saved runs across both the 16 known tasks and the 32 new messages. Of those saved runs, 1,229 expected no database write (such as refusals, policy questions, or requests for missing details) and left the database unchanged. Because no database action happened in those 1,229 runs, the judge should not call their replies "action". Running Jev over those 1,229 replies confirmed the drop: the initial rubric wrongly called 65 of them "action", while the revised rubric called only 1.

Because Claude wrote those 60 labels, agreeing on the 30 held-out examples shows that the rubric edit fixed the “on hold” confusion, not that the judge matches human judgment. Before using a judge for release decisions, have a person who knows your support policy label a separate sample, resolve overlapping categories, and measure false accepts and false rejects. The lab’s labels/README.md lists both samples and verdicts; make part4-agreement recomputes these numbers.

How research tests and improves evaluators

Checking the evaluator by hand works when you review changes yourself. Once a program edits a harness or verifier from evaluator scores, the improver can exploit weak checks or let the grader drift. Five research methods guard that loop:

  1. Attack it, then repair it. BenchJack gives an agent ten public benchmarks to break. It found 219 flaws, and repeated rounds of exploiting and patching cut the share of tasks it could pass without solving them from about 100% to under 10% on four of the benchmarks.
  2. Anchor it to labels the optimizer cannot change. In Who Verifies the Verifier?, an agent improves its own verifier, checked against ten labelled examples. Without those anchors, the verifier decayed until it passed every run. The agents trained with it still looked fine on their task, so the downstream score did not show that the evaluator had broken. LLM-as-a-Judge Is Not an Oracle adds canary tasks that no honest agent can pass. A pass on a canary task shows cheating or a leak in the evaluation, and triggers an audit.
  3. Keep grading material out of the improver’s reach. Harness², from Google Cloud AI Research and Northwestern, improves a harness with the model frozen and no ground-truth feedback to the improver. The research assessment still uses benchmark grading. It runs the proposed harness on the same task as the current one and compares the two runs. Its improver works in a directory that a guard refuses to hand over if it contains a grader’s verdict, rubric or score.
  4. Separate the data for improving from the data for measuring. The lab’s judge was repaired on one labelled sample and measured on another. Beyond Prompts uses separate task sets for repairing a harness, for approving a change, and for the final scorecard.
  5. Grow the metrics from failures. AutoLibra turns people’s comments on agent trajectories into named metrics that a judge can score; on web navigation it found metrics such as error recovery and step efficiency.

An improvement loop: an improver edits the harness, the harness runs the tasks, and the evaluator, marked, grades the runs and sends only scores back. Five numbered marks, keyed below the loop, show where the methods in the list guard it. 1, attack, then repair, on the evaluator. 2, anchor to fixed labels: labelled runs and canary tasks inside the evaluator. 3, keep grading out of reach: a dashed boundary around the improver's workspace, with no verdicts, rubric or labels inside. 4, separate the data, on the tasks the harness runs. 5, grow the metrics: named metrics inside the evaluator.An improvement loop: an improver edits the harness, the harness runs the tasks, and the evaluator, marked, grades the runs and sends only scores back. Five numbered marks, keyed below the loop, show where the methods in the list guard it. 1, attack, then repair, on the evaluator. 2, anchor to fixed labels: labelled runs and canary tasks inside the evaluator. 3, keep grading out of reach: a dashed boundary around the improver's workspace, with no verdicts, rubric or labels inside. 4, separate the data, on the tasks the harness runs. 5, grow the metrics: named metrics inside the evaluator.

The figure places the five methods, numbered as in the list, on a loop in which a program improves the harness. Methods 1, 2 and 5 work inside the evaluator. Method 3 limits what the improver can see, and method 4 splits the tasks the harness runs into separate sets for repairing, approving and the final score.

Comparing designs

A decision weighs several metrics at once: success under the checks you trust, cost, latency, and the use-case metrics. Put them side by side first. On the 32 new messages, with 95% intervals over tasks:

DesignSuccess, obligationsSuccess, with judge$ per taskp95 latency, s
1. Plain agent0.96 (0.91–1.00)0.88 (0.79–0.96)0.00029 (0.00027–0.00032)12.5 (10.9–13.6)
2. Agent with approval middleware0.94 (0.86–1.00)0.86 (0.75–0.94)0.00028 (0.00025–0.00031)9.9 (8.5–11.2)
3. Agent inside a workflow0.94 (0.86–1.00)0.88 (0.78–0.96)0.00027 (0.00024–0.00029)10.8 (8.9–12.4)
4. Router and three agents0.88 (0.75–0.97)0.81 (0.68–0.93)0.00029 (0.00026–0.00032)9.4 (8.3–10.5)
5. Workflow with no agent0.78 (0.62–0.91)0.31 (0.16–0.47)0.00005 (0.00005–0.00005)1.6 (1.5–2.0)

No design leads every column, so no single score can choose between them.

The frontier on two metrics

Pick the two metrics that matter most for the decision and plot every design on them, with an interval on each axis. Here there are two such pairs: success with the judge against cost per task, and success with the judge against p95 latency. The plots use the judge level because it is the strictest of the three. A design is on the frontier when no other design is at least as good on both metrics and better on one. AI Agents That Matter calls this the Pareto frontier and plots it for the accuracy and cost of coding and question-answering agents. Cost frontiers and plots omit any design with a run whose cost is unknown.

Four scatter plots of success with the judge: against dollars per task on the left and 95th-percentile latency on the right, for the 16 known tasks at the top and the 32 new messages at the bottom. Bars are 95% intervals over tasks on both axes, and a line joins the designs on the frontier. On the known tasks the workflow with no agent sits far to the left of every agent design and at least as high; on the new messages it sits far below them.Four scatter plots of success with the judge: against dollars per task on the left and 95th-percentile latency on the right, for the 16 known tasks at the top and the 32 new messages at the bottom. Bars are 95% intervals over tasks on both axes, and a line joins the designs on the frontier. On the known tasks the workflow with no agent sits far to the left of every agent design and at least as high; on the new messages it sits far below them.

On the 16 known tasks, the workflow with no agent is at least as successful as every agent design: 0.94 with the judge, against 0.68 to 0.88. It is cheaper, at $0.00005 per task against $0.00025 to $0.00029. It is also faster, at 2.4 seconds at the 95th percentile against 10.5 to 13.0. The cost and latency intervals are far apart. The design that does nothing is on the frontier too, at zero cost and zero success. A frontier shows which trade-offs are efficient, not which designs are acceptable; the acceptance thresholds decide that.

On the 32 new messages, the frontier runs from the workflow with no agent, at 0.31, up to the agent designs, at 0.81 to 0.88. The workflow’s intervals do not overlap any agent’s on success, cost or p95 latency. Choosing an agent buys 0.50 to 0.57 more success; it costs about six times as much and takes six to eight times as long at the 95th percentile. Among the four agents, only one pair has intervals that do not overlap. The router’s p95 latency (8.3 to 10.5 seconds) is below the plain agent’s (10.9 to 13.6 seconds). Their success and cost intervals all overlap. The plain agent is on the cost frontier because it passed one more run in 160 than the agent inside a workflow. All five designs are on the latency frontier. Another sample could reorder the agents on either plot. Pairing, described next, separates some of them on cost or latency. Graded with the judge, it separates none of them on success; choosing among them needs more tasks or runs.

The lab’s make part4-regrade draws these four plots with harness-evals’ pareto and frontier_svg.

Deciding between two designs

The frontier shows each design with its own interval. To decide whether a candidate should replace the current harness, compare the two directly. harness-evals computes each design’s success rate on each task, subtracts current from candidate, then resamples those paired differences. Both designs face the same tasks, so the comparison accounts for differences in task difficulty. Cost, latency, calls, tokens and the use-case metrics get paired intervals the same way: each resample draws the same tasks for both designs.

Pairing makes the comparison more sensitive than reading two separate intervals. On the new messages, graded on obligations, the success interval is 0.62 to 0.91 for the workflow with no agent and 0.86 to 1.00 for the agent inside a workflow. Those two intervals overlap, yet the interval of their paired difference (−0.29 to −0.04) excludes zero. Paired the same way, the agent inside a workflow is cheaper than the plain agent there, though their separate cost intervals overlap.

The acceptance thresholds come first: a current design that fails them is replaced by a candidate that meets them without adding critical violations. Between two designs that meet the thresholds, only success and cost per task can make the rule adopt the candidate. A cheaper design may give up a little success. The allowed loss is the largest acceptable decrease: 0.02 here means two percentage points, or one additional failure per fifty runs on average. A guardrail limits how much another metric may get worse against the current design, whatever the candidate gains in success or cost. The lab sets one: p95 latency may rise by at most 2 seconds, within the 15-second absolute limit. The remaining metrics are reported with their intervals but do not decide. The rule decides in this order:

  1. Reject if the candidate fails the acceptance thresholds or has more critical violations than the current harness.
  2. Adopt if the current harness fails the thresholds and the candidate meets them.
  3. Reject if the whole 95% interval of candidate minus current lies below −0.02 for success, or past the limit for a guardrail.
  4. Adopt only if every guardrail’s interval stays within its limit and either success improves or cost falls with an acceptable success loss. Improvement means the whole success-difference interval is above zero. The cost route requires a cost-difference interval below zero and a success-difference interval at or above −0.02. A zero-width success interval cannot justify cost-only adoption.
  5. Otherwise the result is inconclusive: keep the current harness, and add tasks or runs.

If both designs pass every run, resampling their equal success rates always gives a difference of zero and an interval of [0, 0]. That interval does not establish a two-point loss limit on future requests. The library therefore keeps cost-only adoption inconclusive whenever the success interval has zero width. A lower cost on this fixed development suite is a measured result; a bound on future success loss needs fresh representative tasks and an uncertainty method that supports that claim.

Before comparing, check that both records use the same complete task set, repetition count, evaluator version and frozen conditions. In the pinned lab, this example compares the workflow with no agent against the agent inside a workflow on the new messages, graded on obligations. Run it with uv run python from the lab’s root:

from harness_evals import compare
from evaluator.__main__ import grades
from evaluator.support import usecase

current = grades("new", "obligations", 3)  # the agent inside a workflow
candidate = grades("new", "obligations", 5)  # the workflow with no agent
result = compare(usecase("obligations"), current, candidate)
print(result.decision, result.reasons)
print(result.deltas)  # each metric: candidate minus current, and its interval

The workflow with no agent costs $0.00022 less per task (interval −0.00025 to −0.00020) and is 9.2 seconds faster at the 95th percentile (−10.9 to −7.1). Rule 1 rejects it before those gains count: it fails the reliability threshold on these messages. The example grades on obligations because the judge’s ambiguous labels have not been reviewed.

Two comparisons from the saved runs show how the checks and tasks change the decision:

  • Design 2, the agent with approval middleware, against design 1, the plain agent, on the 16 known tasks. With the database check, design 2 passes every task (pass^10 of 1.00). Design 1 fails the reliability threshold (pass^10 of 0.81), so rule 2 adopts design 2. Graded on obligations, design 2 is better on 2 tasks and worse on 1, and the interval of the difference runs from −0.18 to +0.19. Design 2 now fails the reliability threshold itself (pass^10 of 0.81), so rule 1 rejects it.
  • Design 5 against design 3, the workflow with no agent against the agent inside a workflow, both graded on obligations. On the 16 known tasks, both pass every run and design 5 costs a fifth as much. The paired success interval is [0, 0], so the cost-only comparison is inconclusive. On the 32 new messages, the comparison in the code above, design 5 scores 0.78 against design 3’s 0.94 (a difference of −0.16, interval −0.29 to −0.04) and fails the pass^5 threshold, so rule 1 rejects it. With the judge added, the gap on new messages widens to −0.56 (0.31 against 0.88, interval −0.73 to −0.38).

Four rows, each the 95% interval of the difference in success between a candidate design and the current one, with a band for the allowed loss of 0.02 left of zero. From the top: design 2 against design 1 with the database check, adopt, and graded on obligations, reject; design 5 against design 3 on the 16 known tasks, inconclusive because the all-pass interval cannot establish the allowed loss, and on the 32 new messages, reject.Four rows, each the 95% interval of the difference in success between a candidate design and the current one, with a band for the allowed loss of 0.02 left of zero. From the top: design 2 against design 1 with the database check, adopt, and graded on obligations, reject; design 5 against design 3 on the 16 known tasks, inconclusive because the all-pass interval cannot establish the allowed loss, and on the 32 new messages, reject.

The figure places each comparison’s interval against zero and the allowed loss. Read the bottom two rows together: the cheaper workflow has an inconclusive result on known tasks and is rejected on new messages.

How many runs and tasks

Repeated runs measure how much one design varies; more tasks measure how far the result carries over to other requests. You need both, in amounts that depend on the size of the difference you want to see:

  • On SWE-bench Verified, a study of 60,000 agent runs found that a single run’s pass rate varied by 2.2 to 6.0 points between repetitions. Standard deviations were 1.0 to 1.8 points even at temperature 0. In that setup, the authors’ power calculation needs about 9 full benchmark repetitions per agent to detect a 2-point difference. It assumes a 1.5-point run-level standard deviation, 80% power and a two-sided 5% threshold; it is not a general nine-trials rule.
  • With 16 tasks, our intervals were about ±0.2. Only differences of a few whole tasks show up.
  • When every run passes, the bootstrap interval shrinks to 1.00–1.00, but uncertainty remains. By the statistical rule of three, nn independent trials with zero failures still leave a 95% upper failure bound of about 3/n3/n, roughly 19% for n=16n = 16. Even that bound assumes a random sample of real traffic; 16 hand-written tasks repeated 10 times are not 160 independent requests.
  • harness-evals requires at least six tasks for an interval-based decision. Its heuristic comes from a two-sided sign test: five nonzero task differences that all point the same way give a p-value of 0.0625, above 0.05. Six is this library’s minimum, not a guarantee that six tasks support a reliable decision.

In our runs, success under the database and obligation checks barely changed across the 10 repetitions. Only the plain agent failed one task in one run that it passed in the other nine. The judge’s readings varied a little more, with one task per agent design changing between runs. Under its database check, Part 3’s single repetition ranked the designs as the 10 repetitions do.

That single repetition’s cost ranking did not hold. In Part 3’s single repetition, the plain agent cost $0.0055 for the 16 tasks and the approval agent $0.0039. Over the 10 repetitions here, the plain agent ranged from $0.0036 to $0.0053. The median was $0.00394 for the plain agent and $0.00408 for the approval agent.

The plain agent’s most expensive repetition was its first, which read 45% of its input from the provider’s prompt cache; its later repetitions read 55% to 69%. A single comparison therefore mixes harness behavior with cache conditions. Interleave the designs’ runs, record the cached share of the input, and compare median costs over many runs.

Evaluation mistakes that skew scores and rankings

A flaw in your evaluation setup or grading can shift scores, flip which design ranks first, or reverse an adoption decision even when the harness stays the same. The table below lists eleven common mistakes engineers make when measuring agents. Some examples come from published benchmarks (LangChain, Anthropic, Darwin Gödel Machine). Others were measured directly in our support-desk runs by comparing weaker checks, a single repetition, or familiar tasks against the full suite to show how far the numbers move.

MistakeExamplePrevention
Accidentally changing a setting along with the harnessLangChain’s score fell from 63.6% to 53.9% when only the reasoning setting changedRecord fixed conditions separately from the changes being tested
Checking only the final databaseA design that does nothing passed half of the support tasksCheck actions and replies; run a do-nothing design first
Letting the harness write its own evidenceA Darwin Gödel Machine version passed its check by removing the log markers the check readGrade externally and restrict access to checks, labels and authoritative records
Trusting an unmeasured judgeThe first judge read 19 of design 2’s 20 “on hold” replies as completed refundsLabel replies before the judge runs; measure it on labels it was not tuned on
Tuning on the tasks you reportThe workflow with no agent scored 0.94 on its 16 tasks and 0.31 on new messages, both with the judgeUse validation tasks to select candidates and reserve fresh tasks for the final assessment
Averaging away a violationNo violation occurred in these runs, but one wrong-customer refund in 200 runs would lower success by only 0.005Report critical violations apart and require zero, as ToolSandbox scores any run that hits a minefield as zero
Treating one run as the resultOne repetition made the plain agent look 40% more expensive than the approval agent; over 10 repetitions their medians were within 4%Repeat and interleave runs; compare median cost; report pass^k and intervals
Deciding on success aloneWith the judge, the plain agent leads on new messages by 1 run in 160. Yet design 3 costs less per task, with the whole paired interval below zero.Report cost, latency, calls and tokens beside success, each with an interval; plot the frontier; set guardrails before comparing
Counting unknown cost as zeroThe router’s decision-model calls came back with no billed cost; counted as zero, they would drop out of design 4’s costCompute it from the price list, or keep it unknown and fail a cost check while any run’s cost is unknown
Mixing infrastructure failures into harness failuresAnthropic’s infrastructure errors fell from 5.8% to 0.5% when only the container resources changedClassify errors; report them apart, still as failures
Comparing results across evaluator versionsDesign 2 scored 1.00 under Part 3’s database check and 0.81 graded on obligationsVersion the checks; compare only runs graded by the same version

The companion code

harness-evals is a Python 3.12+ library with no dependencies outside the standard library. An adapter connects it to a harness: your function runs a list of tasks once and returns one observed record per task. The library’s run function calls each adapter the same number of times, interleaving the designs, and its grading functions check the records. These functions produce the results in this article:

  • summarize reports the metrics above, with intervals for rates, costs, latencies, calls and tokens.
  • pareto and frontier_svg find and draw the frontier.
  • failure_map_svg draws where each harness fails.
  • compare applies the adoption rule. Guardrails are declared on the use case: guardrails={"latency_p95": ("<=", 2.0)} allows a rise of at most 2 seconds.

compare refuses records with different tasks or repetition counts. The harness-evals compare command also validates each design’s loaded records against the manifest: task IDs, split, unique task/trial rows and declared repetitions must match. The manifests fingerprint task IDs, inputs, splits, check names and critical flags, plus the use-case version and frozen settings; from Python, call comparable to compare that metadata. comparable does not compare the evaluator version or the repetition count, and a check whose logic changes under the same name needs a new use-case version.

Install the library in a uv-managed project:

uv add "harness-evals @ git+https://github.com/slavadubrov/harness-evals@03cc80db2fd7039515887baf5c5321c3957139fa"

You describe the use case as tasks with checks and write one adapter per harness. The lab’s live adapter starts the assistant’s runner in a new process, then reads each task’s databases. This excerpt shows where the refund evidence comes from:

def observe(start: Path, final: Path, task_id: str):
    passed = TASKS_BY_ID[task_id].check(start, final).passed
    actions = [
        Action("refund", {"order": r["order_id"],
                          "cents": r["amount_cents"],
                          "refund_id": r["id"]})
        for r in diff(start, final)["refunds"].added
    ]
    ...  # plus the refunds the service still holds for the case

This detects an added refund row even if the harness bypassed the service. If the harness can write to the database files directly, you still need the file and process permissions described above. The adapter handles an agent and a workflow alike because it observes their effects through the same interface.

To recompute the scores from the saved runs, use Python 3.12 and uv. The lab’s Make targets install its pinned dependencies first; no API key or new model call is needed. The commands pin the exact companion revision used here:

git clone https://github.com/slavadubrov/agent-harness-lab-public
cd agent-harness-lab-public
git checkout 19c7712762cca7b4f0af07523f3590ddf8ce8bb9
make part4-regrade       # grades the saved runs; writes RESULTS.md, frontier plots and failure maps
make part4-agreement     # both judge versions and the reply check against the labels
make part4-compare BASE=3 CANDIDATE=5 SPLIT=new
make failures DESIGN=5 SPLIT=new LEVEL=judge        # the checks design 5 fails, task by task
make show DESIGN=2 TASK=refund-over-limit-approved  # one saved run, check by check
make mine                # score your own function on the 16 known tasks

The part4-compare command compares design 5 with design 3 on new messages, graded on obligations. It prints a success difference of −0.163 (95% interval −0.287 to −0.037), along with the paired differences in cost, latency, calls and tokens. It returns reject because design 5 misses the reliability threshold. Add LEVEL=judge to include the saved judge classifications. Regrading reuses the observations and verdicts; it does not independently reproduce the original model behavior.

make show prints one saved run as the evaluator saw it: the customer’s message, what the refund service recorded, the reply and the verdict of every check. make mine runs examples/your_harness.py, a template with no model call and a refund-only API. Replace its function with your own agent or workflow, then update the reported model metadata, calls, tokens and cost. Report real usage when measured and leave unknown values unknown; the rules-only starter’s zero usage does not describe a model call. Supporting address and settings tasks also requires extending its API.

Optional live runs use make part4-live TRIALS=5 with an OpenRouter key in OPENROUTER_API_KEY. This runs all five designs on the 16 known tasks, making paid model calls. The adapter has only been tested with the do-nothing design. At October 2026 prices, the saved agent runs cost about $0.0003 per task and repetition; 5 repetitions of all four agent designs plus the workflow would be roughly $0.10, excluding evaluator calls. Live mode uses obligations and does not run the judge.

What to do next

On your own harness, start with the three checks that changed a result here. Run a design that does nothing: on these tasks it passed 8 of 16 when only the final database was checked. Check the reply as well as the action: the agent with approval middleware (design 2) issued every approved refund and told the customer it was on hold. Score the candidate on messages it was not written for. Here, the workflow with no agent went from 1.00 on its own 16 tasks to 0.78 on 32 new ones under obligations, and to 0.31 once a judge read the replies. Set the thresholds before you read those scores, then compare the two designs with the rule.

The next part will use a program to propose harness changes and this evaluator to score them. Evaluate that search under a fixed budget: count proposals and accepted changes, include proposal and evaluation cost, and report regressions and held-out improvement. Repeat independent searches to assess the procedure, as Beyond Prompts does. Use development failures for edits and validation tasks for selection; keep the evaluator and final test tasks separate. Every candidate must meet the success, cost and latency limits, because retries alone can raise accuracy by spending more (AI Agents That Matter).

This article evaluates harnesses; it does not demonstrate recursive self-improvement. A fixed program searching for better harness code is automated optimization. A stronger recursive claim requires retained changes to affect the process that produces later changes, a distinction Harness² makes between execution and improvement harnesses. That process would need comparison at equal budgets on fresh tasks.