Designing an agent harness: from task to architecture

Viacheslav Dubrov

2026-09-28 · Updated 2026-10-08 · 28 min · AI Agents

An agent harness is the code around a language model that lets it complete a task. It gives the model instructions and data, runs the actions the model asks for, keeps track of progress and decides when the task is finished. This article is for engineers who are adding an agent to their own application. It shows how to design that code from the task: which steps the model chooses, which steps your code fixes in advance, and which rules your code enforces whatever the model says.

Make these decisions before you choose an agent framework, because the answers differ from task to task. A support assistant changes customer accounts, so it needs permission checks and a safe way to retry a payment. A coding agent edits files and runs tests, so it needs a place where its code can run without harming anything else. A document classifier only returns a label, so it may need no tools at all. If all three start from the same loop, memory store and tool set, nobody asks what each task needs.

We will follow these decisions for one example, a customer-support assistant that issues refunds, from its task description to a first implementation. The article builds on Harness Engineering for AI Agents, which describes what a harness is responsible for. Here we choose those responsibilities for one project.

Why the model needs a harness

In our support assistant, the model can propose a refund, but it cannot complete one alone. Code must find the customer’s account, load the order, check that the refund is allowed, call the payment service and record the result. If the payment service times out, code must also decide whether a second attempt is safe. These jobs remain even when the model is very good at choosing the next step.

Some of this code may already exist in your application. LangChain, the agent framework this series uses later, defines a harness broadly: all the code, configuration and run logic around the model. Login, database access, a job queue and an approval step can all fall under that definition. Designing a harness includes deciding how the agent uses them. You do not need to rebuild them inside an agent framework.

A harness can be small or large. The smallest makes one model call, validates the output and returns a result. A larger one lets the model read files, run programs and revise its work over many turns. Both must answer two questions: which parts of the task does the model handle, and how do we check that they were done correctly?

Describe the result before the agent

“Handle refund requests” does not yet say what the assistant must do. Suppose a customer asks for a refund on a delivered order. The run succeeds if it refunds the correct order and amount, explains the result to the customer and changes nothing else on the account. If the request breaks the refund policy, a clear refusal is the correct result. If the assistant cannot tell which order the customer means, it should ask.

A refund, a refusal and a question are three different correct results. If we count every conversation that ends without an error as a success, we cannot tell them apart, and we cannot tell any of them from a wrong answer that ended politely.

A correct refund also depends on who is asking. A customer can type any order number, so an order number in a message does not prove that the order is theirs. The application must supply the logged-in account, and the refund service must check that the order belongs to it. The model can read the request; it cannot prove who the caller is.

We now have the start of a task contract: the changes that count as success, the changes that are forbidden, and the situations in which the assistant must ask a question or hand the case to a person. Some of these requirements cannot be met by the prompt alone. A maximum refund amount must be checked by the service that issues refunds. A response-time target needs a timeout in code. Resuming a case tomorrow needs saved state.

With the task described, we can decide how much of it the model should control.

How much of the procedure do we already know?

If every request follows the same steps (extract an order number, fetch the order, check eligibility, issue an allowed refund and explain the result), we can write those steps in code as a fixed pipeline. A failed check skips the refund. The model handles the language at the start and at the end. It does not need an extra model call to decide whether to fetch an order that every request needs.

An explicit workflow adds branches and waiting states that code controls. A refund above a limit might wait for approval. An address change might be allowed only before the order ships. We can draw these stages as a graph of steps, and code decides which step comes next.

An agent loop lets the model choose each step. The model picks an action, reads the result and picks again until it decides to reply. This helps in a mixed conversation: a customer starts with a delivery complaint, then says the wrong item arrived, then asks for an exchange. The next useful lookup depends on what the assistant has found so far. The cost is that the loop can take more paths, and each one needs testing: the model can make unnecessary calls, repeat a lookup or stop before the task is done.

LangGraph’s workflow and agent guide describes the same split: workflows follow paths set in advance, and agents choose their steps while they run. LangGraph, the graph library underneath LangChain, can build both, so a system built as a graph can still contain an agent.

For our support assistant, one design combines the two. An agent loop gathers information and prepares a proposed refund. A fixed code step then checks the proposal and performs the write. The model can explore the conversation, while code alone decides whether a write happens. If we already know we need this split, we can build the workflow around the agent first.

The other option is to start with a single agent loop that has a step limit, and to put the checks inside the tools themselves. That keeps the first version small while we learn which paths the task needs. Choose the workflow first if approval waits, fixed stages or recovery rules are already requirements.

Splitting the work across several agents is a separate choice, and one agent is often enough. A second agent helps when a subtask needs different context or different permissions, or can run at the same time as the first. Something must then combine their results and settle conflicts between their actions. If a delivery agent and a billing agent can both issue refunds, one order can be refunded twice.

Tool calls or generated code?

After deciding who chooses the next step, we decide how the model asks for an action. It can call a tool: a named function that your code provides and runs on the model’s behalf. Or it can write a short program that your code runs. Who chooses the next step and how the model asks for an action are separate decisions. A fixed workflow can contain a step in which the model writes code, and an open agent loop can use only a few narrow tools.

For the support assistant, a tool such as issue_refund(order_id, amount_cents) fits well. Its schema names the action and its arguments, so the request is explicit. The tool returns a refund ID on success or a structured reason for a rejection. We can test this tool on its own, because the model never writes the refund logic.

Not every task needs tools. A classifier that receives its whole input in the prompt can return a validated label. Add a lookup tool when the task needs data that is not in the prompt. Add a tool that changes data only when the task must change something.

Generated code suits a different kind of work. Suppose the assistant must read a table of orders, group the rows by customer and calculate several totals. With tools, each step is a separate call, and every intermediate result returns to the model. With code, the model writes one Python program that keeps the intermediate values in its own variables, and only the totals come back.

Hugging Face’s smolagents library offers both styles. ToolCallingAgent returns structured tool calls. CodeAgent writes Python that calls the same tools, stores values in variables and combines the results. A code agent still uses tools; it calls them from a program.

Generated code also adds work for us. We must handle syntax errors, limit run time, manage dependencies and decide which files and network addresses the code may reach. Whether fewer model calls repay that work depends on the task and the model. Measure it on your task; code is not better by default.

Several hosted APIs combine both styles: your own business tools plus a built-in code-execution tool that runs in a sandbox, a separate environment with restricted access. OpenAI’s Responses API, the Claude API, the Gemini API and AWS Bedrock AgentCore all offer this.

Anthropic’s programmatic tool calling goes one step further: code in the sandbox can call the tools you allow, and only the results you select return to the model. Anthropic still recommends direct tool calls by default, and the code path for large results and for chains of calls that depend on each other. Some designs use code only, such as smolagents’ CodeAgent and Cloudflare’s Code Mode.

For our refund assistant, I would start with typed business tools. If a later task needs heavy calculation, we can add a code runner that has no access to customer accounts.

A third question is how the application connects to its tools. A local function is enough when one application owns the code. The Model Context Protocol (MCP) is a standard way to serve tools from a separate server, so several applications can share them. MCP defines how the application and the server exchange tool calls. It does not validate arguments or decide who may call a tool: the MCP tools specification leaves those checks to the application and the server.

Put restrictions where actions happen

A tool description tells the model when a tool is useful. It does not decide whether a particular call may run. The code that performs the write must decide that. For a refund, that code checks the logged-in account, the order, the amount, the refund policy and any required approval.

Some checks need interpretation

Some checks cannot be written as ordinary code. Suppose the customer asks, “Can I return this order?” and the agent proposes a refund. The order belongs to the customer, the amount is valid and the return window is open. All the code checks pass, yet the customer may only be asking whether a return is possible, not asking for a refund now. Someone has to read the message to decide.

Ordinary code still checks ownership, amounts and dates. The main model can ask the customer when intent is unclear, or the application can require explicit confirmation before a refund. A separate decision model is another design to evaluate: it answers one narrow question about meaning, such as “Did the customer ask for this refund?” We can test that question on its own, without running the whole conversation. Compare this design with clarification or confirmation before adding another model.

Jev, a decision model from TypeSafe, is one option for that role. It takes the state we supply and typed questions, and returns structured answers. For a yes/no question, it returns the probability that the answer is yes.

I would ask two separate questions: did the customer explicitly ask for the refund, and did a later message withdraw that request? TypeSafe recommends specific questions whose answers are combined in code, and several questions can share one API call. With two questions, a wrong answer shows which of the two judgments failed.

Code then combines those answers with the required checks. If the answers are unclear or conflict, the assistant asks the customer or sends the case to a person. If the Jev call times out, that error needs its own path too. A “yes” from Jev never overrides a failed ownership check or replaces a required confirmation. Its probability is a model output. It does not authorize the refund, and it does not prove that the decision is correct.

The context we give the check decides what it can see. I would supply the proposed action, the customer’s messages in order and the tool results the decision needs, each labelled with its source. The latest message alone could miss a cancellation. The full account history could hide the request among unrelated messages. A small general-purpose model with structured output could also answer these questions. Compare it with Jev on the same examples, for accuracy and cost.

Restrict access independently of model judgments

Text from a customer or a retrieved document can contain instructions aimed at the agent. That text must not be able to grant permissions or replace the application’s policy. If a document tells the agent to export all customer records, the system must lack the permission to do it, however convincing the wording.

Generated code needs its own access limits. If the model can only call reviewed functions with narrow permissions, a general-purpose code sandbox may add little. Those functions still need authorization, input validation, timeouts and safe database queries.

Before we allow generated Python, shell commands or edits to executable files, we must decide what that code can reach. Which directories can it write? Does it need network access? Which credentials can it see? How much CPU time, memory and output can one run use? Running the code in a separate process does not answer these questions by itself.

Hugging Face’s secure execution guide compares a restricted local interpreter with more isolated options such as containers. Each option blocks different things. A container with broad access to the host’s files and powerful credentials can still expose the resources we meant to protect. Test that a forbidden read, a forbidden write and a forbidden network request fail, as well as that a normal program succeeds.

We can leave out the code runner until the task needs one, but its isolation must be ready on the day we enable it. Isolation also cannot stop misuse of an API that we give the code on purpose, so the refund service must still enforce its own rules.

What survives when a run stops halfway through?

Now suppose the assistant sends a refund request, the service creates the refund, and the response is lost on the way back. The model sees a timeout. If it calls again, the customer may get a second refund. If it reports a failure, it tells the customer something that is no longer true.

The fix is to make the refund idempotent: sending the same request twice has the same effect as sending it once. Before sending the refund, the application creates an operation ID and saves it. On a retry it sends the same ID, and the service returns the refund it already made instead of making another. AWS’s guidance on idempotent APIs explains this pattern. If the service does not support it, recovery must first check what happened, or ask a person, before any second attempt.

Saving the conversation does not solve this. The conversation may record that the model asked for a refund but not whether the service completed it. A checkpoint, a saved copy of the agent’s state, lets a stopped run continue. It cannot undo a refund the service already made, and it does not make repeating one safe.

For this assistant, three kinds of state have different jobs. The conversation holds the customer’s request and the facts gathered so far. The order and refund records hold the business state. An execution record lists pending operations and approvals, so a restarted worker can continue safely. They can live in the same database, but one cannot replace another.

Memory across sessions is a separate choice. If the assistant must remember a customer’s preference next week, we need rules for whose preference it is, who can change it and how it is deleted. If every task is independent, we may not need a memory service at all. Each piece of saved text is also a chance to reuse something stale or irrelevant later.

Long conversations raise a similar problem within one session. When a conversation grows too long for the model to use well, we may retrieve parts of it or replace older turns with a summary. A summary that drops an order ID, an approval condition or a pending operation can break the procedure. Keep that operational state in explicit fields, and test what survives a summary before relying on it.

Give the loop a way to stop

An agent loop can always find something else to check, so it needs a stopping rule that covers success, missing information, used-up limits and failure. When a run hits a limit, it must report the true status: the refund is done, was never attempted, or is still uncertain. “The agent stopped” does not tell the customer or the next worker what happened.

It helps to name the ways a run can end before writing the loop. For the support assistant, I would use eight: completed and verified, refused, asked for missing information, handed to a person, blocked by a policy check, stopped by a limit, ended by an infrastructure error, and write outcome unknown.

Record that status separately from whether the run met the task contract. “Completed and verified” confirms the requested write. A refusal, question or handoff can be correct when the task calls for it. The reply, actions and final state must support that verdict; stopping alone does not.

A limit on model calls covers only part of the run. A tool can hang. A retry can repeat work. A summarizer or a decision model such as Jev makes model calls of its own. Set limits on elapsed time, attempts, output size and spending, and count every component that runs. Cancelling a run must also account for requests already sent to an external service.

The model itself adds limits. Check that it produces the required action format on realistic tasks. Compare usable context size, latency, cost, and where task data may be sent. A model you serve yourself and a hosted API can share one interface but differ in capacity and in how they handle data.

Choose an initial model and endpoint, record their settings, and keep them fixed while you evaluate a harness change. Otherwise a faster run may come from a different provider or model, not from the change you meant to test.

Evaluate completion, including the cases that should do nothing

Once the assistant can complete a request, we need to check whether it did the right thing. For the refund example, we can observe three things: what the assistant said, which actions it attempted and what changed in the service. Each one catches failures the others miss.

A reply can say “I refunded your order” while the database is unchanged. A correct refund can come with an unrelated address change that only the record of actions shows. And an unchanged database can mean four different things: a correct refusal, a useful question, a crash or an assistant that did nothing. The final state alone cannot tell these four apart.

The companion demo, described at the end of this article, shows this limit with real numbers. It has twelve support tasks, and seven of them expect no change to the database. An agent that changes nothing therefore passes 7 of 12 state checks. That number describes the checks; it is not a measured success rate. The task definitions compare only the final database, not the reply or the actions in between. A write that is made and then reversed also leaves no trace in that comparison.

For your first evaluation set, include a successful action, a valid refusal, a request with missing information and a failure during execution. For each case, define the expected reply and the allowed actions as well as the final state. A clarification case should require the right question. A refusal case should require a useful explanation. These cases show whether the agent understands the task and whether the application enforces it.

If we add the Jev check, it needs its own test cases. Give it proposed actions whose correct decisions were set by a person in advance: an explicit refund request, a question about eligibility, a withdrawn request and an instruction hidden in a tool result. The last case also tests whether the context we assemble keeps customer messages separate from tool output.

Count the bad actions Jev allows and the valid actions it blocks. After each block, follow the rest of the conversation. Blocking a write helps only if the assistant then asks the customer a question instead of looping or stopping. Jev returns a probability, so the application needs a cut-off that turns it into allow or block. Choose the cut-off on development examples, and fix it before the final comparison on examples you did not tune on. Record the question version, the context supplied, the returned probabilities, the rule applied, any errors and the cost. The recorded demo never had Jev reject a write, so it does not show that Jev helps.

Record enough to diagnose a failure: the task, the model and harness versions, the proposed actions, the execution results, the final state, errors, latency and cost. Keep secrets and unnecessary personal data out of that record.

The agent being tested must not be able to change how it is scored. Give it the task input and the rules it needs, but not the reference answers or the held-out test cases. Keep the evaluator (the code that scores runs) and its records outside the agent’s control. This becomes stricter if a program later edits the harness automatically to raise its score: that program must not be able to edit the checks that score it.

These records also tell us when to add an optional component. Add a router, a first step that sends each request to the right handler, when the recorded runs show that choosing the type of request needs its own stage. Add summaries when long histories cause a specific failure. Test a guard against forbidden actions, including actions it should allow. Required access controls come from the task’s rules; optional components need evidence that they are worth their cost.

Turn the design into a first implementation

For the support assistant, we now have a starting design: an agent loop with limits to interpret varied requests, narrow tools for account operations, checks inside the services that write data, and saved records of pending actions. Nothing so far requires generated code or memory across sessions.

I would build one complete path first. Create a small test account and an expected refund outcome. Implement the lookup and refund tools, including their rejection paths. Connect the model to those tools. Then run the refusal, missing-information and interrupted-write cases. Gaps in the requirements are easier to find now, with a few tasks, than later with many.

A loop without a framework can be enough at this stage. It assembles the context, asks the model for the next action, validates and runs that action, and returns the result to the model. Writing that loop yourself also means writing message conversion, cancellation, state handling and tracing. A framework is worth using when it does that work and still lets you control the decisions in this article.

This series uses LangChain’s create_agent as a starting point, because it provides the model-and-tool loop plus hooks for changing its behavior. It already runs on LangGraph. Moving to an explicit LangGraph workflow later means writing more of the stages around the agent ourselves, not adding a graph from nothing.

Before writing the implementation, I would put the decisions on one page. The first version does not need every capability discussed above. It needs only the parts that complete the task, plus records that show where a run fails.

A useful answer is specific enough to change the implementation. “Needs memory” is vague. “Must resume an approved refund after a worker restart without creating a second refund” tells us what to persist and what failure to test.

Part 2 asks when an explicit workflow handles the task well enough to justify its extra stages. It builds the smallest workflow the tasks need around the agent and compares it with the loop. Later parts keep the evaluator outside the agent’s control and measure how results vary across repeated runs.

Optional: inspect the implementation and its limits

The companion demo is a small, runnable version of the support assistant. It runs in two environments:

• Custom support tasks: twelve tasks, each against a fresh SQLite database of customers and orders.
• τ³-bench retail: eight tasks from τ³-bench, a public benchmark in which a model plays the customer, talks to the agent, and database checks score the result.

This section describes what the demo contains, what it leaves out on purpose, what the results show and how to run it. You can skip it and still use the design process above.

What the demo contains

One LangChain agent chooses the calls. It has five account tools (look_up_account, list_orders, issue_refund, update_address and set_preference) and an MCP server for policy and FAQ lookups. An in-memory checkpointer saves the run state between steps. There is no code runner and no memory across sessions.

Each run reads a spec, a YAML file that names the model and the components to use. The demo records two specs. In base, Schema-Guided Reasoning (SGR) makes the model return a structured next-step object instead of a native tool call. The Jev write guard, the decision-model check described earlier, runs before each write and asks one compound question: would this call break the policy or go beyond what the customer asked for?

The two separate questions earlier in this article, about the request and its withdrawal, are a proposed redesign; the recorded runs did not use them. base adds SGR and Jev in one step, so the custom results cannot show whether SGR or Jev caused a difference.

What the demo leaves out on purpose

• The tools check data integrity but leave business policy to the agent, so an experiment can expose a write the policy forbids. A real service would enforce the policy at the write.
• There is no task-wide limit on time or spending.
• Tool retry applies to every tool, including the three writes, and retried writes have no idempotency key. Retry wraps the Jev guard, so each retried write is also checked again. LangChain’s ToolRetryMiddleware accepts a list of tool names; a real service would limit retries to reads.
• The evaluator runs in the same process as the agent.

These results therefore do not test production enforcement, retry safety or protection of the evidence against tampering.

Results on the custom tasks

Each row is one run of each task. Prices are the endpoint prices recorded with the results. A pass requires the expected final database and no run error. These checks do not verify the reply or all intermediate actions. Cost and latency are means per task; cost includes Jev where it runs.

• With gpt-6-luna, both specs passed all twelve tasks. base cost about 2.2 times as much and took about twice as long.
• Jev allowed every write it checked: six in each of the gpt-6-luna and GLM runs, and four in the MiMo run. It never blocked a write, so these runs do not test whether it stops a bad one.
• MiMo’s five failures were SGR output-format errors. Three left the database unchanged as required, but the run errors still made them failures. There is no MiMo plain run, so we cannot say whether native tool calling would do better.
• Summarization never ran. The largest request in the six recorded runs (these four and the two τ³-bench runs below) had 8,351 input tokens, below the 12,000-token trigger. These results cannot show whether summarization helps. When it does run, LangChain’s summarization middleware replaces the older messages in the agent’s state with one summary message, so the full transcript has to come from the trace.

Results on τ³-bench

τ³-bench scores a run by replaying the agent’s tool calls in a fresh environment, so the benchmark must run the tools itself. The demo’s adapter, the code that connects the agent to τ³-bench, pauses the graph before its tools node, returns the tool calls, and writes the benchmark’s results back before it resumes.

This has two consequences. First, the prompt, tools and policy come from τ³-bench, so these results are not comparable with the custom ones, even with the same spec file. Second, tool retry and Jev never run here: the only difference between base and plain is SGR.

Both runs use openai/gpt-6-luna for the agent and for the simulated customer. Simulation latency includes the simulator’s work. Agent cost and simulation latency are means per task; simulator cost is the total for all eight tasks.

• The eight tasks come from the retail test split and have no natural-language assertions, so database checks alone decide the reward.
• Both specs finished all eight tasks without adapter errors. Every failure was a missing write. base failed tasks 9 and 26. plain handed task 27 to a human instead of completing the exchange.
• A repeat base run, not counted in the six, also passed 6/8 but failed tasks 9 and 17. The failing tasks change between runs, so a one-task difference is no reason to prefer plain. These runs do not measure that variation.

Check the recorded numbers

The saved reports and traces keep the per-task outcomes, token counts, configurations, package versions and recorded prices. For the 393 agent chat calls in the six recorded runs, cost calculated from tokens matched the billed amount in each response. This check excludes Jev and simulated-customer calls. The prices are historical, not a quote for your run.

Run it yourself

Install uv and use the tagged version below. The tests run offline once dependencies are installed. The two environment runs make paid model calls through OpenRouter and need an API key.

git clone https://github.com/slavadubrov/agent-harness-lab-public
cd agent-harness-lab-public
git checkout v0.1.2-a1
cp .env.example .env # set OPENROUTER_API_KEY
make test
make a1-custom SPEC=harness/spec/plain.yaml
make a1-tau3 SPEC=harness/spec/plain.yaml

Each run replaces the results under reports/article-a1/. Use git diff to compare them with the committed baseline. The Makefile installs from the frozen uv.lock.

To try another model or tool set, copy a spec under harness/spec/, change those fields and pass its path through SPEC. Provider and custom-component settings accept free-form keys, so validation cannot catch every nested typo.