Long-Running AI Agent Runtime: Sessions and Checkpoints

Viacheslav Dubrov

2026-05-26 · Updated 2026-08-01 · 37 min · AI Agents

An agent run may last for hours, while its worker process may restart at any time. The model still chooses the next action, but the runtime must preserve state, control execution, and recover from a failure in the middle of a tool call. This post defines the runtime boundary. Part 6 then opens the one component in it that decides — the harness, which is where memory, tool contracts, and permission checks stop being three topics and become one program.

What is an AI agent runtime?

An AI agent runtime is the infrastructure layer that keeps a tool-using agent alive, isolated, observable, and resumable after the model call ends. It owns session state, tool execution, checkpoints, secrets, traces, cost limits, and deployment shape. The model chooses the next action; the runtime determines where that action executes, how it is recorded, and how the run resumes after failure. Whether the action is allowed is decided by the harness. The harness is not a storage layer; it is the program that stands on them, and it sits in the table below because you have to place it too.

Four of those five — session, sandbox, checkpoint, trace — store state or confine execution. The harness is the one that decides, and it is where the memory, tool-contract, and permission decisions all land. This article treats it as one box and describes what it stands on. Part 6 opens the box, and re-sorts the same five into the one component that decides and the four it runs on.

Long runs break stateless process assumptions

A stateless chat endpoint can keep request state in one process and discard it after the response. A long agent run crosses worker restarts, deploys, context resets, and approval pauses. The worker process can no longer be the source of truth.

The OpenAI Codex team reports how long these runs get in its harness engineering write-up:

“We regularly see single Codex runs work on a single task for upwards of six hours (often while the humans are sleeping).”

Anthropic’s engineering team describes the corresponding state problem in Effective harnesses for long-running agents:

“The core challenge of long-running agents is that they must work in discrete sessions, and each new session begins with no memory of what came before.”

Both observations imply the same runtime design: persist state outside the worker and make workers replaceable.

The session must live outside the worker process. A durable store records model calls, tool results, and approvals so another worker can resume at the last safe point after a crash. Checkpoints also let the runtime start a fresh model session when the context window fills, without replaying the full history. In Anthropic’s phrasing, harness instances become disposable and restartable; durable state lives elsewhere.

Five primitives to place before you ship

Anthropic’s Scaling Managed Agents write-up provides a useful vocabulary for five runtime responsibilities. The harness drives the agent forward, while the session records what it did and the sandbox executes commands. The checkpoint gives the next worker a resume point; the trace preserves evidence for later debugging. An implementation can merge components, but the responsibilities and failure boundaries still need names.

Session. An append-only log of everything that happened: model calls, tool calls, results, errors, and approvals.

The word is overloaded, so pin down three spans called a session. A thread is a user’s conversation across days. It is the longest-lived of the three, and LangGraph tracks it with a thread_id.

A model session is the shortest: one continuous stretch of model context. Compaction — the step that summarizes the window so the work can keep going — extends a model session rather than ending one. A restart or a deliberate fresh start ends it. Part 6 uses “model session” in that sense.

In this article, “session” means the durable log of one run. It sits between the two: many model sessions write into one log, and one thread accumulates many logs. Recovery is wake(sessionId) → getSession(id) → resume from last event.

In LangGraph, recovery uses a thread_id plus a Postgres checkpointer (see LangGraph persistence). The OpenAI Agents SDK ships ten built-in session backends, including SQLiteSession, RedisSession, SQLAlchemySession, MongoDBSession, and EncryptedSession (see the Sessions docs).

Harness. The orchestration loop, and the only primitive here that makes decisions. It assembles the prompt from memory, calls the model, checks the proposed tool call against its permission rules, dispatches what it allows, writes results back into the session, applies retry rules, and decides whether the task is finished. Every one of those steps encodes an assumption about what the model cannot do by itself. Anthropic makes that point directly — it is quoted in the failure-modes section below, which is mostly about what happens when those assumptions go stale.

OpenAI’s Codex team calls this harness engineering: writing software still takes engineering effort, but more of it now goes into the scaffolding than into the code itself. LangGraph’s CompiledStateGraph, LangChain’s Deep Agents and its create_deep_agent entry point, and Claude Code itself are all harnesses in this sense.

Sandbox. The isolated execution environment where commands actually run. The OpenAI Agents SDK sandbox concepts page draws the line cleanly:

“The outer runtime still owns approvals, tracing, handoffs, and resume bookkeeping. The sandbox session owns commands, file changes, and environment isolation.”

“Outer runtime” there means the harness together with its state stores. In this series’ vocabulary, approvals and handoffs are harness decisions (Part 4 and Part 6); tracing and resume bookkeeping are the session and checkpoint primitives.

Sandboxes differ in how long they live and what they remember between runs. The simplest shape is fresh ephemeral: spin one up for a single task, destroy it when the task ends, and pay the cold-start cost on every run.

Persistent paused sandboxes keep the filesystem and a memory snapshot between runs. The next resume can avoid a full boot. Snapshot or fork branches a copy-on-write image from a prepared parent, so many tasks share installed dependencies and warm caches without sharing their writable state.

Per-worktree sandboxes give each task its own workspace and observability stack. Separate logs, metrics, and traces let you debug one run without its state bleeding into another. The provider table later in this article compares cold-start and persistence behavior.

Checkpoint. Resumable state.

LangGraph’s PostgresSaver writes a Checkpoint at every super-step boundary. A super-step is one round of the graph, either a single node or a batch that ran in parallel. Per-task writes go to checkpoint_writes, so successful node outputs are not recomputed when a sibling fails.

A checkpoint is a plain dict (v, id, ts, channel_values, channel_versions, versions_seen, updated_channels). LangGraph serializes it with its msgpack-based JsonPlusSerializer rather than JSON. datetime, set, Decimal, and dataclasses all round-trip. The format is documented on the langgraph-checkpoint-postgres PyPI page and the LangGraph checkpoints reference.

StateSnapshot is the separate, richer view that graph.get_state() builds on top of a checkpoint. That is the object whose .values the debug bundle later dumps.

Trace. The replay and debug surface. Every model call, tool call, and sub-agent step becomes a span with timing, inputs, outputs, token counts, and cost. When a six-hour run fails, the trace is what you read to figure out what went wrong. The terminal output from the run is long gone by then. OpenTelemetry’s GenAI semantic conventions standardize the attribute names (which model, which provider, how many tokens, which conversation, which workflow). For an OTLP-compatible destination that supports these conventions, the same instrumentation can export the trace to systems such as Tempo, Jaeger, Honeycomb, or LangSmith, although backend adapters or destination-specific configuration may still be required.

Policy and secrets cut across the primitives

Two boundaries cut across all five primitives. They’re the runtime version of the security argument from Part 4. The permission decision itself belongs to the harness; what follows here is where the machinery that enforces and feeds it physically sits.

Permission enforcement

The permission ladder from Part 4 needs somewhere to run. The check fires before every tool call and decides whether it goes through. Two patterns are common in production. Deep Agents lets each subagent declare which file paths it can read or write, and the middleware blocks anything outside that declaration. Anthropic Managed Agents routes every tool call through a Model Context Protocol (MCP) proxy, so the proxy enforces permissions instead of the agent code. When a sensitive call needs human approval, LangGraph’s interrupt() and Deep Agents’ approval hook pause the graph until a person says yes.

Secret broker

The model should not see long-lived secrets, and the sandbox usually should not either. The Managed Agents pattern is the one to copy:

“For Git, we use each repository’s access token to clone the repo during sandbox initialization and wire it into the local git remote. Git push and pull work from inside the sandbox without the agent ever handling the token itself. For custom tools, we support MCP and store OAuth tokens in a secure vault. Claude calls MCP tools via a dedicated proxy; this proxy takes in a token associated with the session. … The harness is never made aware of any credentials.”

In the market-analyst-agent reference stack — a small LangGraph agent that fetches market data and writes an analyst report, built across this series — the MCP sidecar holds the data-provider API keys and exposes only the tool surface to the LangGraph worker. In the local compose file both containers read the same .env, which is a development shortcut rather than the pattern. In production the sidecar’s environment comes from a secret store — Docker secrets, or HashiCorp Vault — that the worker cannot read. The worker then calls the tool without ever holding the credential behind it.

Sanity-check the placement

A practical sanity check is to write down every component and which of the five primitives it implements. Postgres might cover session and checkpoint. The worker container is the harness. A service such as Daytona, Modal, or E2B provides the sandbox, while Tempo or LangSmith stores the trace.

Then inspect coupled failures. If two primitives live in the same process, one crash takes both down. If they share a credential, one leak crosses both boundaries. Common examples are a worker that also owns trace durability or a sidecar token that also unlocks the checkpoint database.

Production AI agent runtime failure modes

The runtime manages retries, restores prior work, isolates workspaces, and enforces budgets. As runs stretch across workers and context windows, failures shift toward state, duplicate side effects, sandbox drift, and budget overruns.

The failures fall into four groups:

• Quality-of-output failures: the agent declares victory before the work is actually done, forgets what it did across a context-window reset, or trusts its own self-evaluation and ships broken output.
• Cost-control failures: the agent gets stuck in a retry loop, or spends through a token or tool-call budget without producing anything useful.
• State and crash failures: workspaces drift because one run touches files another run owns, tool calls fire more than once because retries replay them, or work is lost when a worker dies between events.
• Context-window failures: the model summarizes and quits early because it thinks it is running out of room, even when the window still has headroom.

The table maps each failure to a mitigation, the basis for the recommendation, and the runtime hook that enforces it. Model-specific behavior can change, so treat vendor observations as prompts to re-test the assumption rather than permanent rules.

Two ideas sit behind most of those rows. Anthropic, on harness staleness in Harness design for long-running application development:

“Every component in a harness encodes an assumption about what the model can’t do on its own, and those assumptions are worth stress testing, both because they may be incorrect, and because they can quickly go stale as models improve.”

Vercel, on the related problem of too many tools encoding too many assumptions, in We removed 80% of our agent’s tools:

“We deleted most of it and stripped the agent down to a single tool: execute arbitrary bash commands. We call this a file system agent.”

The quote describes the bash core; the agent Vercel shipped kept two tools, ExecuteCommand and ExecuteSQL, down from fifteen. Part 3 covers the full before-and-after. Their reported result across five representative queries: success went from 4/5 to 5/5, and the worst case dropped from 724 s / 100 steps / 145,463 tokens (failed) to 141 s / 19 steps / 67,483 tokens (succeeded). That worst-case row is the dramatic one; averaged across the five queries the token saving was 37%. The lesson is not “delete your tools.” It’s that every primitive in your runtime, including the tool surface, has a half-life. Re-test the assumption when the model changes.

Cognition saw the same moving target on session length with Sonnet 4.5. In Rebuilding Devin for Claude Sonnet 4.5 they describe a model that proactively writes SUMMARY.md / CHANGELOG.md as it senses context exhaustion but underestimates how many tokens it has left. Their fix was to enable the 1M-token context and cap usage at 200k so the model still believes it has headroom. That was a beta flag when they wrote it.

Anthropic’s current context-window documentation as of August 2026 still lists Sonnet 4.5 at 200k. The 1M window ships as the default, with no beta header, on Opus 4.6 and later and Sonnet 4.6 and later. The cap is the part worth watching. It exists only because Sonnet 4.5 misjudges its remaining context, so the day a model stops doing that, the cap stops being a fix and becomes an artificial ceiling. Model generations have turned over more than once since Cognition wrote that post; re-check the numbers against the current model list before carrying any of this forward.

OpenAI’s harness team has the one-line version: “Humans steer. Agents execute.” When something fails, the useful question is which capability is missing, and how to make that capability both legible and enforceable for the agent.

The healthy run lifecycle

A well-behaved run is boring. It is a chain of small recoverable steps, and each step writes its result to durable storage before the next one starts.

Writing each result before the next step starts is what contains crash damage. A failure loses only the step in flight, and the next worker resumes from the last completed step instead of restarting the full request.

1. Boot from either a fresh session or a resumed one. On resume, mount the workspace from its last known state, read any progress files the previous attempt left behind (PROGRESS.md, feature-list.json), and load the last checkpoint from the database. This is where the harness hands the agent everything the previous worker had in memory before it died.
2. Plan before any tool calls fire. Write down what “done” looks like, how much the run is allowed to spend, which tools the agent can invoke, and what should stop the run early. These plan values become runtime checks; without them, execution has nothing to push back against.
3. Execute one tool call at a time. The harness’s permission check decides whether to allow it, then dispatches it, captures the result, and writes one event into the session log. One step, one event. A crash between events is recoverable because the log, not the worker’s memory, is the source of truth.
4. Checkpoint at super-step boundaries, or after every event in a simpler harness. Persist the graph state, the workspace diff, and references to any artifacts produced. This checkpoint is what step 1 reads on the next resume. If the checkpoint is missing or stale, recovery degrades into replaying the full session log from scratch, which is much slower.
5. Evaluate against the artifacts when the agent thinks it is done: tests, a fresh-context evaluator, schema validation, browser checks. If the check passes, the run exits successfully. If it fails, the run resumes from the last clean checkpoint with the failure message added to context and tries again.

There is no step in that list that requires the agent to remember anything between runs. The state lives in the session and the checkpoint, and the agent reads it back on each resume.

Any tool with side effects needs an idempotency key derived from the session ID and tool-call ID, stored before the side effect fires. send_email(session_id, tool_call_id, message_hash). create_pr(session_id, tool_call_id, branch_name). charge_customer(session_id, tool_call_id, invoice_id). At-least-once execution is the default in queues and workflow engines, so the duplicate will happen. If a tool call can cause real harm when repeated and you cannot deduplicate it on a key, the tool is not ready for agents.

Evaluation should include evidence outside the producing context. A fresh-context evaluator reduces shared-context bias, while tests, lints, browser checks, and schema validation provide deterministic evidence. The check can return pass, fail, or needs_human. For code agents, the reviewer may be another model session with read-only tools. For data and report agents, combine deterministic validation with a reviewer model where judgment is still required.

Eleven AI agent deployment patterns and what decides between them

Once the five primitives are named, the question is which deployment shape runs them. By “shape” I mean an arrangement of those primitives: where the harness lives, where state persists, and what kind of sandbox runs the work. A shape is a wiring decision, not a vendor pick. The chart below shows where each shape is comfortable on the run-length axis. The text after it walks through what decides between them.

If you only read one of the eleven, read shape 2: queue + worker + checkpoint DB. It is the default I recommend for most teams, the shape used by the reference repo, and the skeleton most other shapes vary on: queue → worker → durable state, with the sandbox source, harness owner, or state engine swapped out. Reading shape 2 first makes the rest faster to scan.

The chart compares shapes by run length. The matrix below compares them by ownership: each outlined cell names the component that provides that primitive.

1. SDK inside an app server (synchronous, request scoped)

The original shape. The agent SDK runs inside a request handler. Good for sub-30-second tasks, demos, and internal tools. Bad for anything an HTTP client might disconnect from. Cloud Run’s HTTP timeout maxes out at 60 minutes, and any web-tier panic kills the run. The SDK is the harness, the web process also acts as the sandbox, and state usually lives in process memory unless you explicitly push it elsewhere. Do not use this for multi-hour work.

2. Queue + worker + checkpoint DB

The default I recommend for most teams, and the production shape used in market-analyst-agent: a Python worker with a PostgreSQL checkpointer, Redis Streams (or RabbitMQ) for the inbound queue, and an MCP sidecar for tools. Good for 10-minute to multi-hour runs with idempotent steps. The local runner can bypass the queue for synchronous development, but the queue is part of the production shape once you need async submission and backpressure.

The app accepts a request, creates a session row, pushes a job, and returns a run ID. The worker pulls the job, runs the harness, writes checkpoints, streams status, and stores artifacts as it goes. Postgres survives, workers are cattle, and queue depth gives you backpressure. Spot/Preemptible compute works as long as the checkpointer finishes writing to disk before it reports success.

In this shape, the worker is the harness. Its container and per-thread workspace provide an execution boundary, but untrusted code still needs a hardened sandbox or VM. Postgres owns session and checkpoint state. Traces go through OpenTelemetry to whatever observability stack you run.

3. Durable workflow engine (Temporal-style)

Agent orchestration code runs inside a Temporal Workflow; model calls and tool calls run as Activities. Workflow state lives in an event-history log backed by Cassandra, MySQL, or Postgres, so state replays cleanly across deploys. The Temporal × OpenAI Agents SDK integration, generally available since March 2026, ships an OpenAIAgentsPlugin and an activity_as_tool helper, and the agentic sandboxes write-up describes forking a running agent onto a different sandbox provider mid-conversation. Idle workflows consume zero compute. The caveats are real: realtime agents are unsupported and streaming is still marked experimental, and LocalShellTool and ComputerTool are disabled because they don’t fit a distributed model.

Use this shape when the run has real waiting points: human approvals, external callbacks, long sleeps, retries with business rules, deploy windows. A human approval becomes a durable sleep that consumes no compute, not a polling loop that does.

The Workflow code is the harness. The sandbox usually lives outside Temporal and is called from Activities. Session and checkpoint state collapse into Temporal’s event-history log, while trace visibility comes from Temporal UI plus OpenTelemetry spans on each Activity.

4. Sandbox provider per session

A newer shape. Every agent run gets its own microVM or container from a sandbox-as-a-service provider. The harness lives somewhere durable; the sandbox is the disposable execution environment.

Cold starts here are end-to-end provisioning, not raw boot: E2B’s ~150 ms sits on top of the ~125 ms Firecracker boot that Part 4 quotes for the hypervisor itself. The table combines the E2B vs Daytona comparison, Daytona’s sandboxes documentation and fork/snapshot changelog, Modal’s sandboxes and cold-start guides, the Runloop AWS Marketplace listing, and Vercel Sandbox pricing.

Daytona records a parent-child link for each independent fork, which preserves the lineage of derived sandboxes. OpenAI’s Codex harness uses the per-worktree variant: “Codex works on a fully isolated version of that app, including its logs and metrics, which get torn down once that task is complete.”

Reach for this shape when the agent runs untrusted code, browser automation, tests, or package installs. The trade-off is cost and provider coupling, both higher than running shared workers.

The provider owns the sandbox and nothing else. Harness, session, checkpoint, and trace stay on your side, usually wired as the queue + worker shape from #2.

5. Anthropic Managed Agents (hosted harness)

Anthropic launched Managed Agents in public beta on April 8, 2026, behind the managed-agents-2026-04-01 beta header. The service provides a hosted session, harness, sandbox, and vault-backed MCP proxy. wake(sessionId) can initialize the harness on a new worker without losing durable session state.

Anthropic bills Managed Agents at standard token rates plus $0.08 per session-hour. Billing is millisecond-granular and applies only while the session status is “running”; idle time is free. A runaway retry loop therefore adds session-hour cost on top of its token cost.

Read the caveats. The Batch API discount does not apply (“Sessions are stateful and interactive. There is no batch mode.”). Managed Agents is not available through AWS Bedrock or Google Vertex AI. Within the beta, MCP tunnels and agent “dreaming” sit behind a further research preview you have to request access to; multi-agent coordination and rubric-graded self-evaluation are documented parts of the beta. Lock-in is high: you trade harness freedom for not running the loop yourself.

Anthropic hosts all five primitives: session, harness, sandbox, checkpoint, and trace. You hand over the runtime and get the outputs.

6. LangChain Deep Agents Deploy (managed open harness)

deepagents deploy packages a deepagents.toml into a LangSmith Deployment with durable execution, memory, multi-tenancy, human-in-the-loop, observability, sandboxed code execution, and scheduled runs. Cloud, hybrid, and self-hosted deployment modes are supported. Sandbox providers (LangSmith Sandboxes, Daytona, Modal, Runloop, or custom) are switchable via a single config value. State lives in a virtual filesystem with pluggable backends; memory is scoped to user, assistant, or both. Lock-in is lower than Managed Agents: the harness is MIT-licensed, instructions use the open AGENTS.md standard, and agents are exposed via MCP, the A2A (Agent2Agent) protocol, and Agent Protocol. See LangChain’s runtime-behind-production-deep-agents write-up.

All five primitives are hosted by default, but each is config-swappable. The sandbox sits behind one config value. Session and checkpoint live on a virtual filesystem with pluggable backends. Trace goes to LangSmith.

7. Google Cloud Run service or job

Cloud Run has two different runtime modes, and which one fits depends on how the agent is invoked. Services are HTTP-bound and scale to zero between requests; the harness runs as a request handler that returns when the run is done. Jobs run to completion without an HTTP entrypoint; the harness runs as a one-shot worker that exits when the task finishes. Both can host the harness, but neither holds state across runs. Sessions and checkpoints have to live in Postgres, Spanner, or a similar external store.

The hard limits are very different between the two. Cloud Run service request timeout: default 300 s, max 3,600 s (60 min). WebSockets get the same timeout. Cloud Run jobs: default 10 min per task, max 168 h (7 days); for tasks using GPUs, max 1 hour. Services scale to zero unless you enable always-on CPU; jobs do not have HTTP and do not autoscale.

Use a service for synchronous runs up to 60 minutes. Use a job for longer one-shot or async work. Cloud Run Jobs can keep a task alive for days, but they do not give you durable replay across deploys, version changes, or worker replacement. Above 7 days, do not use Cloud Run.

Cloud Run hosts the harness. Session and checkpoint state live in Postgres, Spanner, or another external store, and traces can flow through Cloud Logging and OpenTelemetry. The service container is an execution environment; add a separate sandbox when the agent runs untrusted code.

8. AWS Lambda (why it is the wrong tool)

Lambda’s maximum function timeout is 900 s (15 minutes), hard. If API Gateway fronts the function, the integration limit depends on the API type. HTTP APIs allow 30 seconds; REST integrations default to 29 seconds, while Regional and private REST APIs can configure a longer timeout. None of those paths turns Lambda into an hours-long worker. A long-running harness still needs external state and re-invocation, which recreates the queue + worker shape. Use Lambda for bounded tool calls, such as file fetches or S3 uploads, invoked by a longer-running orchestrator. Do not put the orchestrator there.

At most, Lambda holds one tool call inside its 15-minute cap. The harness, session, checkpoint, sandbox, and trace all have to live somewhere else.

9. AWS ECS / Fargate task per run

Fargate documents no hard cap on task runtime, unlike Lambda. Fargate throttling quotas allow a launch burst of 100 and refill at 20 per second, with separate on-demand and spot budgets. ECS service quotas cap services using AWS Cloud Map discovery at 1,000 tasks per service and EC2-backed clusters at 5,000 container instances.

Fargate requires awsvpc mode, so every task gets a network interface and private IP. That shape fits VPC-internal data access. Fargate Spot adds interruption risk, and durability remains your responsibility because the platform has no Temporal-style replay.

Fargate hosts the harness and gives each run its own task. That separates workspaces and task credentials, but it is not a complete sandbox for hostile code by itself. Session, checkpoint, and trace go to external services such as RDS or DynamoDB plus CloudWatch/X-Ray.

10. Kubernetes Job or namespace per session

Good when you already operate Kubernetes and want sandbox-per-session with cluster-wide controls. Bad when you need sub-second startup, because pulling the container image and initializing the pod takes too long on a cold start. The pattern is one Job per agent run, with activeDeadlineSeconds, a PersistentVolumeClaim for the workspace, and a sidecar for the MCP server. Crash recovery is yours to build. Adopting Kubernetes just to host agents is expensive in configuration overhead and operational burden. Only worth it if you already run K8s for other reasons.

Kubernetes hosts the harness and per-run execution environment, usually as one Job and sometimes with a dedicated namespace. Strong isolation still depends on runtime class, network policy, pod security, and the underlying container or VM boundary. Session and checkpoint state live in an external database or on a PersistentVolumeClaim.

11. Local Docker Compose (dev only)

The reference for the next section. The point of this shape is that it mirrors the production topology one-for-one (same primitives, same network shape) while running on a single box. What it does not mirror is the isolation: one shared workspace mount, one Postgres, no hardened sandbox, and no separate failure domains between the worker and its state. Do not ship anything shaped like this.

Compose mirrors shape #2 on a single host. Postgres holds session and checkpoint state, and the worker container is the harness. The shared workspace mount is convenient for development but does not isolate untrusted runs. The optional OpenTelemetry stack records traces.

Reference stack: Docker Compose

The reference topology, used in slavadubrov/market-analyst-agent, is a LangGraph worker, a Postgres checkpointer, Qdrant for retrieval, an MCP sidecar, a Redis queue for async production-like runs, and an optional Prometheus / Grafana / Loki / Tempo / OTel observability stack. In local compose, Redis is optional only because the synchronous runner can call the worker directly. docker compose up brings the core topology up locally; the MCP sidecar and the observability stack are opt-in profiles (--profile mcp, --profile observability).

The one piece worth showing inline is the canonical LangGraph wiring. It is an illustrative excerpt, not a repository-runnable example. Running it requires langgraph, langgraph-checkpoint-postgres, and psycopg[binary,pool], a reachable PostgreSQL database with permission to create the checkpointer tables, POSTGRES_PASSWORD, and a previously built StateGraph in builder; see LangGraph’s Postgres checkpointer setup.

import os
from urllib.parse import quote

from langgraph.checkpoint.postgres import PostgresSaver

password = quote(os.environ["POSTGRES_PASSWORD"], safe="")
DB_URI = f"postgresql://agent:{password}@postgres:5432/agent"
# `builder` is your StateGraph, already built
session_id = "session-123"
with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
    checkpointer.setup()  # creates tables on first run
    graph = builder.compile(checkpointer=checkpointer)
    result = graph.invoke(
        {"messages": [{"role": "user", "content": "Continue the task"}]},
        {"configurable": {"thread_id": session_id}},
    )

Observability that survives the run

Short request handlers are easy to debug: when something fails, you read the response and the live log. Long-running agents do not have that luxury. By the time a six-hour run fails, the interesting event happened five hours ago, the live terminal output is gone, and the worker that produced it has been replaced. Nobody is going to reconstruct the run from memory. So you debug from durable artifacts that were written while the run was still alive.

Production stacks tend to cover four kinds of artifact, in two groups. Two of them you read after the run is over, for postmortems and replay: a queryable event log of every step, and OpenTelemetry traces of where time and tokens went. Two of them you read during the run. One is a live tail of what the agent is producing in the workspace. The other is a per-worktree observability stack that the agent itself can query while it is still working.

Structured event log (read after the run)

Every model call, tool call, result, error, and approval written to durable storage, keyed by session ID and timestamp. Once the run ends, you query it like a normal database table. Addy Osmani sets the bar plainly in Long-running Agents: “If you can’t reconstruct what the agent did in the last 24 hours from durable storage, what you have is a long-running shell script that happens to call an LLM, not a long-running agent.”

OpenTelemetry GenAI traces (read after the run)

The same kind of step-by-step data is emitted as spans using the standard attributes from the gen_ai.* semantic conventions: model name, provider, input and output token counts, conversation ID, and workflow name. The conventions are still at Development stability.

In 2026 they moved out of OpenTelemetry’s main semantic-conventions repository into their own GenAI semantic conventions repo. The attribute names are usable to instrument against, but pin whichever revision you validated rather than a main-repo version number. Provider-specific fields live in subnamespaces (anthropic.*, openai.*) keyed off gen_ai.provider.name. The reason to use the standard is portability: on OTLP-compatible destinations that support these conventions, switching backends may not require re-instrumenting the code, though backend adapters or destination-specific configuration can still be required.

Tool-call timeline plus workspace diffs (read during the run)

The fastest way to know what an agent is doing right now is to tail what it is producing in the workspace, not to grep through a session log. Anthropic’s Harness Primitives for Long-Running Claude Agents quick-start ships a two-pane watch loop for this: watch -n 5 'git log --oneline -8' shows the latest commits the agent has made, and watch -n 5 'find screenshots -name "*.png" | tail -5' shows the latest screenshots it has taken. Two terminal panes refreshing every five seconds is enough to tell whether a run is making real progress or spinning.

Ephemeral stack per worktree (read by the agent itself, during the run)

Per OpenAI’s harness post: “Logs, metrics, and traces are exposed to Codex via a local observability stack that’s ephemeral for any given worktree.” Each agent worktree gets its own short-lived Loki + Prometheus + Tempo, scoped to that run alone. The agent queries it while it works. That is what lets a prompt like “no span in these four user journeys exceeds two seconds” become something the agent can verify directly, rather than something it has to guess at.

(The fresh-context evaluator from the failure-modes table reads these artifacts to decide “done.” It belongs to evaluation, not observability; see § healthy run lifecycle. It depends on every surface above.)

A minimal self-hosted observability stack

For something like market-analyst-agent:

1. OpenTelemetry Collector with the GenAI Normalizer Processor (contrib, alpha) for supported GenAI attributes. Use the generic Attributes or Transform processors to filter or rewrite gen_ai.* fields.
2. Tempo (or Jaeger) for traces, keyed by gen_ai.conversation.id / thread_id.
3. Loki for structured event-log entries.
4. Prometheus for gen_ai.client.token.usage, gen_ai.client.operation.duration, and gen_ai.client.operation.time_to_first_chunk — the gen_ai.server.* metrics come from the model server, so you only get them if you host the weights (see the GenAI metrics conventions).
5. Grafana dashboards keyed on gen_ai.agent.name and gen_ai.request.model.

Hosted alternatives (pick one, not three):

• LangSmith: native LangGraph integration; also the deployment target for Deep Agents Deploy.
• Braintrust: strongest fit if eval-first regression suites are the priority.
• Arize Phoenix: OSS, native to OTLP (the OpenTelemetry wire protocol), pairs with OpenInference instrumentation.
• OpenAI’s tracing dashboard: automatic when you use the OpenAI Agents SDK or its Temporal integration.
• Anthropic’s Claude tracing: for sessions running inside Managed Agents.

Instrument the LangGraph node

This is an illustrative excerpt and is skipped by the repository’s example runner. It assumes the LangGraph node already has an active OpenTelemetry span, the current thread_id, and a provider response usage object with input_tokens and output_tokens; tracer setup, export configuration, and provider-specific usage mapping are outside the snippet.

# In the LangGraph node, around the model call:
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.provider.name", "anthropic")
span.set_attribute("gen_ai.request.model", "<your-model-id>")
span.set_attribute("gen_ai.response.model", "<your-model-id>")
span.set_attribute("gen_ai.conversation.id", thread_id)
span.set_attribute("gen_ai.agent.name", "market-analyst")
span.set_attribute("gen_ai.workflow.name", "research_then_write")
span.set_attribute("gen_ai.usage.input_tokens", usage.input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", usage.output_tokens)

Attribute names taken verbatim from the OpenTelemetry GenAI semantic conventions registry.

Three queries worth having on a dashboard

# Loki: token usage per agent over 1h
sum by (gen_ai_agent_name) (
    rate({service_name="market-analyst-agent"} | json | unwrap gen_ai_usage_output_tokens [1h])
)

# PromQL: p95 model latency per model
histogram_quantile(0.95,
    sum by (le, gen_ai_request_model) (
        rate(gen_ai_client_operation_duration_bucket[5m])
    )
)

# TraceQL: long-running tool calls
{ span.gen_ai.operation.name = "execute_tool" && duration > 30s }

The debug bundle pattern

When a run fails, the worker should drop /workspaces/${THREAD_ID}/_debug/ containing the artifacts you would ask for in a postmortem:

• session.jsonl: full event-log dump from the PostgresSaver (checkpointer.list({"configurable": {"thread_id": ...}})).
• last_state.json: StateSnapshot.values from the last successful super-step.
• trace.json: OTLP-exported spans for the run.
• tool_calls.csv: (ts, tool, input_hash, latency_ms, status, error).
• workspace.tar.zst: the workspace directory plus git diff against the initializer commit.
• screenshots/*.png: what the agent saw.
• PROGRESS.md, feature-list.json, and any other agent-authored progress files.
• env.txt: image tags, model version, harness commit SHA.

This bundle gives a human or reviewer agent enough evidence to reconstruct the failure. “The agent got stuck” is vague. An illustrative report is concrete: session s_123 spent 71 percent of its tokens repeating three commands after npm install failed.

Picking the right shape: a decision guide

Most of the comparison above collapses into a handful of decisions.

Start with run length

Use run length as the first filter:

• Under 30 seconds, idempotent: request-lifecycle SDK in an app server.
• 30 s to 60 min: queue + worker + checkpoint DB.
• 60 min to 24 h: same queue + worker, or a Cloud Run Job for one-shot work. Use a durable workflow engine if you also need versioning and replay.
• More than 24 h, must survive deploys: durable workflow engine (Temporal-style). Cloud Run Jobs can hold long work up to their task limit, but they do not provide replay semantics.
• Multi-day reinforcement-learning training loops: K8s Job + volume + Temporal.

After that coarse filter, check side effects, recovery, replay, isolation, data location, and the team that will operate the system.

Platform fit by use case

The matrix is dense, and no single green cell decides the architecture; the yellow cells, where a platform supports something only with a caveat, are usually the ones that decide it. Broad workload coverage is useful, but it does not show data residency, replay semantics, provider dependence, operational maturity, or the cost of moving state later.

Deep Agents Deploy is the only column in the matrix with no red or yellow cell: short synchronous runs, multi-hour batch, sandbox forking, GPU work, and lowest lock-in all come out green. That makes it a candidate when one platform has to serve every workload you have. That breadth comes with a shorter production track record than a queue + worker + Postgres stack. Treat the green cells as capability claims to validate, then compare the operational constraints that the matrix cannot encode.

Anthropic Managed Agents either fits your workload entirely or not at all. The product has two hard constraints: it is hosted-only and Claude-only. If your workload satisfies both — Claude is already the model you want, and you would rather not operate a harness yourself — Managed Agents is a strong fit. An internal coding agent running in two-to-six-hour bursts is the shape that fits best, and it removes a large chunk of platform work from your team. If either constraint fails because you need a non-Claude model or self-hosted compliance, Managed Agents does not fit. No amount of configuration changes that.

The pricing is worth modeling before you commit, not after. The session-hour line is $0.08/hour on top of standard token costs. If a single session ran continuously, that is about$58/month per session. At 100 sessions running continuously, it is about $5,800/month _before tokens_. Multiply$0.08 by your expected concurrent-session hours, add it to your token bill, and compare against what a queue + worker stack on your own infra would cost. Migrating off Managed Agents later is a re-platforming exercise, not a config change.

Hosted harness vs owned harness

The distinction here is about who operates the harness, not who wrote its code. Hosted means the vendor runs the harness loop in their infrastructure and you call an API. Owned means you run the loop on your own infrastructure, even if the harness code itself came from a vendor.

LangChain shows up on both sides of this line, which trips people up. They ship LangGraph, an MIT-licensed library you self-host (owned), and Deep Agents Deploy, a managed product that runs a Deep Agents harness on LangSmith Deployment in its default cloud mode (hosted). Same company, two different operational models. What you are choosing is who runs the loop, not whose logo is on the library. (Deep Agents Deploy also has a self-hosted mode for teams that want the harness ergonomics without the cloud component; that mode lands in the owned bucket.)

Choose a hosted harness when its model support, data boundary, recovery behavior, and extension points already fit. Choose an owned harness when those constraints are requirements you expect to change. Migration between the two changes state, observability, and execution boundaries, so test the exit path before production data depends on it.

Hosted sandbox vs an owned execution environment

Pick a hosted sandbox when the provider’s isolation, pause/resume, or fork semantics match the threat model and startup budget. Docker or Fargate can fit trusted internal workloads that need VPC access or strict data residency, but a standard container is not a sufficient boundary for hostile code. Part 4 walks the isolation menu for that case.

State stores: Git, DB, and object storage side by side

Long-running agents usually use three state stores at once because each store owns a different artifact.

Git stores workspace state: the code, documents, and progress files the agent changes. Each commit gives the harness a stable recovery point and the next session a compact history.

The checkpoint database stores graph state: what was decided, which nodes ran, which results returned, and what should run next. The artifact store holds large final outputs such as PDFs, Parquet files, and screenshots. Those artifacts do not belong in Git or the checkpoint database.

When to use git as state

Use git when the workload is code-shaped (multi-file edits, refactors, app generation) or document-shaped enough that file history matters. The pattern is simple: create a run branch, make an initializer commit, then commit at meaningful boundaries: after setup, after each feature, after tests pass, after the final cleanup. Store the latest workspace commit SHA next to the checkpoint row. On resume, the next worker checks out the branch, reads git log --oneline -8, inspects git status and the latest diff, then reads PROGRESS.md or whatever handoff file the previous session wrote.

That makes git a recovery surface for the artifact under edit, not a replacement for the checkpoint DB. Git can answer two questions: what changed, and which version passed tests. It cannot tell the harness which graph node should run next, which tool call is waiting for approval, or which retry already used its idempotency key. Anthropic’s harness uses initializer commits plus per-feature commits as the source of truth for workspace recovery; the model reads git log --oneline -8 to recover state. Skip git when the work product is a single conversational answer. The overhead does not pay off.

When to use DB checkpointing

Use PostgresSaver-style checkpointing when the agent has a graph structure with multiple nodes whose intermediate state matters (planner → researcher → writer → verifier). The reference repo uses this for exactly that reason. Do not put terabyte-scale workspace artifacts in the checkpoint; those go to object storage.

When to use an artifact store (S3 / GCS)

Use object storage when:

• the output is larger than the checkpoint database should carry;
• downstream consumers need a URL-addressable artifact without going through the agent; or
• the deliverable and the run state have different retention windows.

For example, you might drop the session log after 30 days but keep the final report for years. Key the layout by (thread_id, checkpoint_id, artifact_name) so the producing run remains reconstructable.

When to add human approval gates

Add gates when the tool call is destructive and irreversible (DB writes, money movement, sending external comms), when the tool call exits the agent’s blast radius (production deploys, customer-facing publishes), or when regulators require review. LangGraph’s interrupt() and Deep Agents’ approval middleware both have built-in support for these gates. Part 4 covered why these gates are a permission concern, not a prompt concern.

A practical production checklist

Before a long-running agent ships, answer these questions in concrete infrastructure terms.

1. Which store owns session events and checkpoints?
2. What happens if the worker dies halfway through a tool call?
3. Can one run corrupt another run’s workspace?
4. Which actions require approval?
5. Can the model or sandbox read raw credentials?
6. Which tool calls can safely retry?
7. Where is the per-run cost cap enforced?
8. What fresh-context evaluator decides “done”?
9. Where do final outputs live after the sandbox is gone?
10. Can we explain a failed run tomorrow without re-running it?

If the answer to any of these is “the prompt tells the agent to be careful,” the system is not deployed yet. It’s still a demo.

The next layer is the harness loop

This runtime can keep a run alive and recoverable, but durability does not prove the work is correct. Part 6, Harness Engineering for AI Agents, opens the harness primitive from the table above: how a trace tells you which of several failures you actually have, where retry and stop rules live, what a handoff must preserve, and how an external acceptance check decides that a run is done. It is also the last post in the series.

References

Engineering write-ups

• OpenAI, Harness engineering: leveraging Codex in an agent-first world.
• Anthropic Engineering, Effective harnesses for long-running agents.
• Anthropic Engineering, Harness design for long-running application development.
• Anthropic Engineering, Scaling Managed Agents: Decoupling the brain from the hands, April 8, 2026.
• Cognition AI, Rebuilding Devin for Claude Sonnet 4.5: Lessons and Challenges.
• Vercel, We removed 80% of our agent’s tools.
• Addy Osmani, Long-running Agents.

LangGraph and Deep Agents

• LangGraph docs, Persistence.
• LangGraph reference, Checkpoints.
• langgraph-checkpoint-postgres on PyPI.
• LangChain docs, Deep Agents overview.
• LangChain blog, The runtime behind production Deep Agents.

OpenAI Agents SDK

• OpenAI Agents SDK, Sessions.
• OpenAI Agents SDK, Sandbox concepts.

Temporal

• Temporal blog, Introducing Temporal and agentic sandboxes: the OpenAI Agents SDK.
• Temporal blog, Production-ready agents with the OpenAI Agents SDK + Temporal.
• Temporal × OpenAI Agents SDK contrib README (temporalio/sdk-python).

Anthropic platform

• Anthropic, Claude platform pricing: Managed Agents session-hour rates.
• anthropics/cwc-long-running-agents: Code with Claude 2026 take-home with evaluator subagent and progress-file patterns.

Sandbox providers

• ZenML, E2B vs Daytona: sandbox comparison for platform engineers.
• Daytona docs, Sandboxes.
• Daytona changelog, Sandbox fork and snapshot endpoints.
• Modal docs, Sandboxes.
• Modal docs, Cold start guide.
• Runloop on AWS Marketplace.
• Vercel Sandbox pricing and limits.

Cloud platform timeouts and quotas

• Google Cloud, Configure request timeout for services.
• Google Cloud, Using WebSockets.
• Google Cloud, Set task timeout for jobs.
• AWS, Configure Lambda function timeout.
• AWS, Lambda quotas.
• AWS, Fargate throttling quotas.
• AWS, ECS service quotas and API throttling limits.

Observability

• OpenTelemetry, Semantic conventions for generative AI systems.
• OpenTelemetry, Gen AI attributes registry.
• OpenTelemetry, Semantic conventions for GenAI agent and framework spans.
• OpenTelemetry, Semantic conventions for generative AI metrics.

The Market Analyst Agent code (LangGraph worker, Postgres checkpointer, Qdrant memory, MCP sidecar, and the Docker Compose topology described above) is on GitHub.