Enterprise RAG Challenge 3: Lessons from Public Entries

Viacheslav Dubrov

2026-01-11 · Updated 2026-07-15 · 9 min · AI Agents

Enterprise RAG Challenge 3 (ERC3) asked agents to complete business tasks against a simulated company API. The frozen prize leaderboard is unusually useful because many entrants published more than a score: architecture, model mix, cost, and failure notes.

I reviewed those public descriptions to answer a narrower question: which design choices recurred in strong submissions, and which of them are useful outside this benchmark?

By the end, you should be able to turn these observations into design hypotheses for your own agent traces, then test them against your task mix and failure costs.

What is the Enterprise RAG Challenge?

The Enterprise RAG Challenge 3 is a large-scale, crowdsourced research project that tests how autonomous AI agents handle complex business tasks. Unlike static benchmarks, ERC3 runs on the Agentic Enterprise Simulation (AGES), a discrete-event simulation that exposes a realistic enterprise API.

What the benchmark tests

Through AGES, agents work inside a fake company that has:

• Employee profiles with specific skills and departments
• Projects with team assignments and customer relationships
• Corporate wiki with business rules and permission hierarchies
• Time tracking and financial operations

Each task spins up an isolated simulation. The company wiki is shared, but operational records vary by task, so an agent cannot solve the suite by memorizing one company state.

Read the scores as a snapshot

ERC3 now exposes both a frozen competition leaderboard and a public benchmark that continued to receive runs after the event. Those pages answer different questions. The figures below describe the prize leaderboard at the competition cutoff, not later best-performing sessions:

The live benchmark page can show higher scores because it includes later runs. That makes the frozen leaderboard the right source for claims about what won the competition.

Types of tasks

The tasks span several skill areas:

• Multi-hop reasoning, such as matching employee skills to project assignments.
• Permission validation, such as blocking unauthorized salary changes or data access.
• Ambiguous queries, including multilingual and paraphrased requests.
• Strict output compliance, including mandatory entity links in responses.

What the submissions actually suggest

The public write-ups do not support a clean verdict such as “multi-agent beats single-agent.” The fourth-place prize entry was explicitly a simple single-agent design. They do support four narrower observations:

1. Decomposition was useful when it isolated a known failure boundary. Teams separated permission checks, step validation, code execution, or response formatting, not arbitrary “agent roles.”
2. Validation moved closer to irreversible actions. Several systems checked permissions before execution, reviewed individual steps, or guarded the final response.
3. Trace-driven iteration mattered. The prize winner converted failed runs into prompt revisions through an automated loop; other teams documented similarly concrete tool and prompt fixes.
4. Context policy was an architectural choice. Teams tried distillation, preloading, retrieval, and history compression. Their own reports disagree on whether compression helped, so there is no universal recipe.

Five informative approaches

These are not the top five in rank order. I selected them because their public descriptions expose five distinct ways to build the system: automated prompt revision, specialist stages, per-step validation, response guards, and plan-execute isolation. Where I interpret why a design helped, I label that interpretation rather than treating it as a leaderboard finding.

1. Evolutionary prompt engineering (Team VZS9FL / @aostrikov)

The highest-scoring approach automated prompt engineering through a self-improvement loop.

Instead of hand-tuning the production prompt, the team built a three-agent loop that turned failed traces into candidate revisions.

Three-agent pipeline:

The production prompt was the 80th auto-generated version. The team describes the loop as analyzing failed tasks, proposing causes, and deciding which suggestions to incorporate. The leaderboard establishes the final score and iteration count. It does not isolate how much of the gain came from automation rather than the models, tools, or accumulated benchmark feedback.

Stack: claude-opus-4.5 with Anthropic Python SDK and native Tool Use.

2. Multi-agent sequential pipeline (Team Lcnxuy / @andrey_aiweapps)

This submission built a sequential workflow in which specialist components owned security checks, context extraction, execution, and entity-link formatting.

The documented components:

1. Security Gate Agent: Pre-execution check that validates permissions against wiki rules before the main loop runs.
2. Context Extraction Agent: Pulls the critical rules out of massive prompts and preloads user, project, and customer data.
3. Execution Agent: ReAct-style planning with 5 internal phases (Identity → Threat Detection → Info Gathering → Access Validation → Execution).
4. LinkGeneratorAgent: Embedded inside the response tool, parses context to include the required entity links.

The LinkGeneratorAgent is the most transferable part. Putting it inside the response tool makes a benchmark requirement (mandatory entity links) a property of the interface rather than one more instruction the execution model can forget.

Stack: atomic-agents and instructor frameworks with gpt-5.1-codex-max, gpt-4.1, and claude-sonnet-4.5.

3. Schema-guided reasoning with step validation (Team NLN7Dw / Ilia Ris)

This team paired SGR with fast inference and a validator on every proposed step. The design makes revision cheap: reject a flawed step before it becomes a tool call, then ask the main flow to rework it with the validator’s comments.

Key components:

The team reported running gpt-oss-120b on Cerebras at up to roughly 3,000 tokens per second. The team paired validation with high-throughput inference, which may have reduced its latency cost. The public result does not isolate that effect.

Stack: gpt-oss-120b on Cerebras, with a customized SGR NextStep implementation.

4. Enricher and guard system (Team J8Gvbi / @mishka)

This submission added non-blocking hints and a tiered guard system to an SGR base. As API responses came back, enrichers inspected them and added operational guidance to later context.

More than 20 enrichers inspected API responses and injected contextual hints:

RoleEnricher: "You are LEAD of this project, proceed with update."
PaginationHintEnricher: "next_offset=5 means MORE results! MUST paginate."

Three-mode guard system:

Hybrid RAG wiki: Three search streams (regex, semantic, and keyword) covered different query shapes against the company wiki.

Stack: qwen/qwen3-235b-a22b-2507 on the LangChain SGR framework.

5. Plan-execute REPL (Team key_concept_parallel)

This architecture put a hard wall between planning and execution and used a code-generating loop. It appeared on the broader Ultimate leaderboard, not the frozen prize top five. Its public description is still useful because it shows a different form of decomposition: isolation by execution phase rather than by business role.

Different models handled different jobs: one planned, one wrote Python, and a separate decision model chose what to do after each step.

Multi-model setup:

The step completion REPL:

1. Planner creates a high-level step.
2. Code-gen model works in a fresh model context and writes a Python script for it.
3. Script executes in a task-scoped REPL whose variables persist across steps.
4. Decision model looks at the result and picks: continue, abort, or replan.

The replan path is the reusable idea. When a step partially fails, the decision model can preserve completed work and rewrite only the remaining plan.

Patterns that recurred across submissions

The implementations differed, but several engineering concerns appeared repeatedly in the public descriptions.

Context management was explicit

No team could hand the model every rule, record, and prior step without making a policy choice. The interesting difference was where each system filtered information.

Trade-off: NLN7Dw compressed older turns, while f1Uixf reported that history compression hurt its experiments and kept the full conversation instead. Treat compression as a measured choice, not a default.

Guardrails were placed at different failure boundaries

Several teams placed checks before, during, or after the main loop. These mechanisms addressed different risks and should not be collapsed into one generic “critic agent.”

Tool wrappers

Several teams built abstraction layers around the raw API:

• Auto-pagination: Wrappers loop through every page and return the complete dataset.
• Fuzzy normalization: “Willingness to travel” gets translated to the will_travel API field.
• Specialized reasoning tools: think, plan, and critic tools for controlled deliberation.

Failure modes and the structural fixes teams reported

The write-ups repeatedly mention failures at API and policy boundaries. The most reusable fixes moved the requirement into code or into a dedicated validation step:

A practical adoption order

ERC3 is one simulated company, not a general agent ablation study. Use it as a source of design hypotheses, then test those hypotheses against your own traces. A sensible adoption order is:

1. Make API correctness deterministic first. Auto-paginate list endpoints, normalize fuzzy fields, validate schemas, and generate required links inside the response tool.
2. Add checks at real risk boundaries. Verify identity and permission before mutation; validate a step before execution only when the extra model call catches failures worth its cost.
3. Write down a context policy. Decide what is preloaded, retrieved, compressed, or kept verbatim. Measure the policy by task slice rather than token count alone.
4. Turn failed traces into regression cases. Classify the failure, change one mechanism, and rerun the affected slice. Automate prompt revision only after that loop is trustworthy.
5. Decompose when ownership becomes clearer. A separate component is justified when it can own a constraint, use a different model or tool, or be tested independently, not simply because “multi-agent” sounds more capable.

Across these write-ups, the reliable submissions made hidden operational requirements visible in tools, validators, and evaluation loops.

References

• Enterprise RAG Challenge 3 - Official Site
• ERC3 Leaderboards
• Benchmark: erc3-prod
• Release Notes