Enterprise RAG Challenge 3: Lessons from Public Entries

Viacheslav Dubrov

2026-01-11 · Updated 2026-10-08 · 13 min · AI Agents

Enterprise RAG Challenge 3 (ERC3) asked agents to complete business tasks against a simulated company API. The frozen prize leaderboard is unusually useful because many entrants published more than a score: architecture, model mix, cost, and failure notes.

I reviewed those public descriptions to answer a narrower question: which design choices recurred in strong submissions, and which of them are useful outside this benchmark?

This article is for engineers building agents that call business APIs. You will see five designs from the public write-ups, the problems each one solved (permissions, pagination, missing links, context size), and an order in which to try them in your own agent.

What is the Enterprise RAG Challenge?

The Enterprise RAG Challenge 3 is a large-scale, crowdsourced research project that tests how autonomous AI agents handle complex business tasks. Unlike static benchmarks, ERC3 runs on the Agentic Enterprise Simulation (AGES), a discrete-event simulation that exposes a realistic enterprise API.

What the benchmark tests

Through AGES, agents work inside a fake company that has:

• Employee profiles with specific skills and departments
• Projects with team assignments and customer relationships
• Corporate wiki with business rules and permission hierarchies
• Time tracking and financial operations

Each task spins up an isolated simulation. The company wiki is shared, but operational records vary by task, so an agent cannot solve the suite by memorizing one company state.

Read the scores as a snapshot

ERC3 now exposes both a frozen competition leaderboard and a public benchmark that continued to receive runs after the event. Those pages answer different questions. The figures below describe the prize leaderboard at the competition cutoff, not later best-performing sessions:

The live benchmark page can show higher scores because it includes later runs. That makes the frozen leaderboard the right source for claims about what won the competition. The Ultimate leaderboard takes each account’s best run without the competition cutoff, so its scores are not comparable with prize scores.

Types of tasks

The tasks span several skill areas:

• Multi-hop reasoning, such as matching employee skills to project assignments.
• Permission validation, such as blocking unauthorized salary changes or data access.
• Ambiguous queries, including multilingual and paraphrased requests.
• Strict output compliance, including mandatory entity links in responses.

What the submissions suggest

The public write-ups do not support a clean verdict such as “multi-agent beats single-agent.” The fourth-place prize entry, MMzXeM, was one agent with tools mapped 1:1 to the API endpoints, an ask_wiki tool, and a 320-token system prompt. The write-ups do support four narrower observations:

1. Teams split out a component when it owned one requirement that agents kept failing: a permission check, step validation, code execution, or entity links. They did not split by arbitrary “agent roles.”
2. Validation moved closer to irreversible actions. Several systems checked permissions before execution, reviewed individual steps, or guarded the final response.
3. Failed traces drove the changes. The prize winner converted failed runs into prompt revisions through an automated loop; other teams documented similarly concrete tool and prompt fixes.
4. Context policy was an architectural choice. Teams tried distillation, preloading, retrieval, and history compression. Their own reports disagree on whether compression helped, so there is no universal recipe.

Five informative approaches

These are not the top five in rank order. I selected them because their public descriptions expose five distinct ways to build the system: automated prompt revision, specialist stages, per-step validation, response guards, and plan-execute isolation.

1. Evolutionary prompt engineering (Team VZS9FL / @aostrikov)

The highest-scoring approach automated prompt engineering through a self-improvement loop.

Instead of hand-tuning the production prompt, the team built a three-agent loop that turned failed traces into candidate revisions.

Three-agent pipeline:

The production prompt was the 80th auto-generated version. The team describes the loop as analyzing failed tasks, proposing causes, and deciding which suggestions to incorporate. The leaderboard establishes the final score and iteration count. It does not isolate how much of the gain came from automation rather than the models, tools, or accumulated benchmark feedback.

Stack: claude-opus-4.5 with Anthropic Python SDK and native Tool Use.

2. Multi-agent sequential pipeline (Team Lcnxuy / @andrey_aiweapps)

This submission built a sequential workflow in which specialist components owned security checks, context extraction, execution, and entity-link formatting.

The documented components:

1. Security Gate Agent: Pre-execution check that validates permissions against wiki rules before the main loop runs.
2. Context Extraction Agent: Pulls the critical rules out of system prompts longer than 500 lines and preloads user, project, and customer data.
3. Execution Agent: ReAct-style planning with 5 internal phases (Identity → Threat Detection → Info Gathering → Access Validation → Execution).
4. LinkGeneratorAgent: Embedded inside the response tool, parses context to include the required entity links.

The team’s write-up says the LinkGeneratorAgent was there to prevent missing-link failures: answers with the right text but without the required entity links. It is the most transferable part. Putting it inside the response tool makes a benchmark requirement (mandatory entity links) a property of the interface rather than one more instruction the execution model can forget.

Stack: atomic-agents and instructor frameworks with gpt-5.1-codex-max, gpt-4.1, and claude-sonnet-4.5.

3. Schema-guided reasoning with step validation (Team NLN7Dw / Ilia Ris)

Schema-guided reasoning (SGR) makes the model fill a fixed output schema at each step, for example its reasoning, a plan, and the next tool call, so the program can read and check each field. The SGR article on this blog shows how to enforce such a schema with vLLM. This team paired SGR with fast inference and a validator on every proposed step. The design makes revision cheap: reject a flawed step before it becomes a tool call, then ask the main flow to rework it with the validator’s comments.

Key components:

The team reported running gpt-oss-120b on Cerebras at up to roughly 3,000 tokens per second. The team paired validation with high-throughput inference, which may have reduced its latency cost. The public result does not isolate that effect.

Stack: a modified NextStep agent, the SGR sample agent published for ERC3.

4. Enricher and guard system (Team J8Gvbi / @mishka)

This submission added non-blocking hints and a tiered guard system to an SGR base. As API responses came back, enrichers inspected them and added operational guidance to later context. The guards checked the agent’s response before it was submitted.

More than 20 enrichers inspected API responses and injected contextual hints:

RoleEnricher: "You are LEAD of this project, proceed with update."
PaginationHintEnricher: "next_offset=5 means MORE results! MUST paginate."

Three-mode guard system:

Hybrid RAG wiki: The team searched a local copy of the company wiki three ways: regex for structured patterns such as salary|privacy, sentence-transformer embeddings for semantic matches, and token overlap as a keyword fallback. My reading of why three: regex finds exact field names and terms, and embeddings find rules worded differently from the question.

Stack: qwen/qwen3-235b-a22b-2507 on the LangChain SGR framework.

5. Plan-execute REPL (Team 5qsp7i / key_concept_parallel)

This architecture kept planning and code writing in separate model calls. Generated Python runs in a REPL (read-eval-print loop): a Python session that keeps its variables between steps. It appeared on the broader Ultimate leaderboard, not the frozen prize top five. Its public description is still useful because it shows a different form of decomposition: isolation by execution phase rather than by business role.

Different models handled different jobs: one planned, one wrote Python, and a separate decision model chose what to do after each step.

Multi-model setup:

The step completion REPL:

1. Planner creates a high-level step.
2. Code-gen model works in a fresh model context and writes a Python script for it.
3. Script executes in a task-scoped REPL whose variables persist across steps.
4. Decision model looks at the result and picks one of four outcomes: continue with the next step, finish because the task is done, abort because it failed, or replan.

The replan path is the reusable idea. When a step result shows the plan no longer fits, the decision model can keep completed work and rewrite only the remaining plan steps. In the diagram, done and failed share one exit to the final response.

Patterns that recurred across submissions

The implementations differed, but several engineering concerns appeared repeatedly in the public descriptions.

Context management was explicit

No team could hand the model every rule, record, and prior step without making a policy choice. The systems differed in where they filtered information: before the run, at retrieval, or in the history. The diagram splits the model’s context into four parts and shows which policy controls each part, when it runs, and which teams used it.

Trade-off: NLN7Dw compressed older turns, while f1Uixf reported that history compression hurt its experiments and kept the full conversation instead. Treat compression as a measured choice, not a default. Strong entries also differed on prompt size. Kc7F2N (3rd prize) kept its own instructions minimal, relied on distilled wiki rules, and built its search tools on embeddings. MMzXeM (4th prize) kept the main prompt to 320 tokens and put the whole wiki behind its ask_wiki tool, which relied on context caching to stay cheap.

Checks before, during, and after the main loop

Several teams placed checks before, during, or after the main loop. These checks need different data at different stages. A shared implementation can perform them if it receives the right inputs at each stage. A critic invoked only for the final response cannot prevent an earlier unauthorized tool call. The diagram places each check on a timeline from task to grader and shows what it sees and the failure it stops.

Tool wrappers

Several teams changed the tool layer the model sees instead of adding instructions about the raw API:

• Auto-pagination: NLN7Dw removed pagination from every tool. One function loops through all pages and returns the full list.
• Fuzzy normalization: J8Gvbi’s tool parsers map human wording to API names, so “Willingness to travel” becomes the will_travel field.
• Schema cleanup: f1Uixf removed unused fields from tool schemas and expanded the parameter descriptions. NLN7Dw rebuilt Req_LogTimeEntry, the one SDK tool whose fields came in a different order, because that order confused the model.
• Reasoning tools: xoDvsa (7th prize, 0.515) added plan, think, and critic tools to a plain ReAct agent. The critic tool returns structured output with dedicated reasoning fields.

A wrapper hides pagination from the model: the agent asks for a list once, and list_all follows next_offset until it is -1, the terminal value used in the official ERC3 sample. The code below is a sketch of step 1 of the adoption order at the end of this article, not any team’s code. respond builds the entity links from IDs the agent fetched and rejects an ID it never saw, so every link comes from code and an invented ID fails before the grader sees it.

# Stand-in for the benchmark API: 12 projects, 5 per page.
PROJECTS = [{"id": f"proj_{i}", "name": f"Project {i}"} for i in range(12)]

def call_api(endpoint: str, offset: int = 0, limit: int = 5) -> dict:
    page = PROJECTS[offset : offset + limit]
    more = offset + limit < len(PROJECTS)
    return {"items": page, "next_offset": offset + limit if more else -1}

def list_all(endpoint: str) -> list[dict]:
    """Follow next_offset until the API returns the terminal value -1."""
    items, offset = [], 0
    while offset != -1:
        page = call_api(endpoint, offset=offset)
        items.extend(page["items"])
        offset = page["next_offset"]
    return items

def respond(text: str, entity_ids: list[str], fetched: list[dict]) -> dict:
    """Build required links from IDs; refuse IDs the agent never fetched."""
    known = {row["id"] for row in fetched}
    unknown = [i for i in entity_ids if i not in known]
    if unknown:
        raise ValueError(f"unknown entity ids: {unknown}")
    return {"message": text, "links": [{"id": i} for i in entity_ids]}

projects = list_all("/projects/list")
assert len(projects) == 12  # three pages, one list
print(respond("You lead two projects.", ["proj_3", "proj_11"], projects))

The agent never sees a page boundary, and proj_11 from the third page is a valid link.

Failure modes and the structural fixes teams reported

The write-ups repeatedly mention failures where the agent calls the API or applies a company policy. The most reusable fixes moved the requirement into code or into a dedicated validation step:

Two reports add detail to the loop row. f1Uixf saw tool-calling loops with GPT-5 but not with the smaller Qwen3 model. NLN7Dw set its turn limit too low, and 5 of 103 tasks did not finish, so a turn limit needs headroom.

A practical adoption order

ERC3 is one simulated company, not a general agent ablation study. Use it as a source of design hypotheses, then test those hypotheses against your own traces. A sensible adoption order is:

1. Make API correctness deterministic first. Auto-paginate list endpoints, normalize fuzzy fields, validate schemas, and generate required links inside the response tool.
2. Add checks where a mistake is expensive. Verify identity and permission before mutation; validate a step before execution only when the extra model call catches failures worth its cost.
3. Write down a context policy. Decide what is preloaded, retrieved, compressed, or kept verbatim. Measure the policy by task slice rather than token count alone.
4. Turn failed traces into regression cases. Classify the failure, change one mechanism, and rerun the affected slice. Automate prompt revision only after that loop is trustworthy.
5. Decompose when ownership becomes clearer. A separate component is justified when it can own a constraint, use a different model or tool, or be tested independently, not simply because “multi-agent” sounds more capable.

The figure below sketches what each failure looks like in a trace and points to the fix to try first. The numbers match the steps above.

References

• Enterprise RAG Challenge 3 - Official Site
• ERC3 Leaderboards
• Benchmark: erc3-prod
• Schema-Guided Reasoning (SGR)
• Release Notes