AI Engineering Labs

Code for the articles on this site. Each card says what the project measured and what you need to run it.

Labs for the series

• Market Analyst Agent

  Series: Engineering the Agentic Stack

  A LangGraph research agent that fetches market data and prepares reports. The series uses it as the reference implementation for reasoning, memory, tools, guardians, and runtime.

  Result: No benchmark. The tests check tool calls on synthetic market data and how the guardian routes trades for approval.

  Run it: uv run pytest tests/test_tools.py uses synthetic data and needs no model. A full analysis needs Python 3.13+, Docker, and model and search API keys, and it makes paid model calls.

  Repository · Part 1

• Agent Harness Lab

  Series: Building and Evaluating Agent Harnesses

  A LangChain support agent that handles refunds, delivery questions, and account changes. Spec files turn harness components on and off, so two configurations run on the same tasks.

  Result: With gpt-6-luna, both specs passed 12 of 12 custom state checks. The spec with Schema-Guided Reasoning and the Jev write guard cost about 2.2 times as much and took about twice as long. On eight τ³-bench retail tasks the guarded spec passed 6 and the plain spec passed 7, but the failing tasks changed between runs, so that gap does not favor either spec.

  Run it: make test runs offline. make a1-custom and make a1-tau3 call models through OpenRouter and need an API key.

  Repository · Article

• Recursive Harness Engineering Lab

  Series: Recursive Harness Engineering

  An LLM proposes changes to a memory tool, deterministic Python tests each change on 20 synthetic stories, and the test results decide the next proposal.

  Result: Three campaigns started at 13 of 20 and all reached 20 of 20 with the same three patches, using 12 model calls for about USD 0.04. A sweep of all 384 configurations ran in under a second and found 16 that pass, so this search did not need an LLM.

  Run it: lab verify and lab serve check the recorded study without an API key. A new campaign needs OPENAI_API_KEY and the --live flag.

  Repository · Article

More labs

• RAG Evals Demo

  Notebooks and modules that compute retrieval, reranking, and generation metrics on the SciFact corpus, plus a sweep over chunking, embedding, and LLM settings. Qdrant runs embedded, so you do not need Docker.

  Result: No SciFact benchmark is reported. Unit tests pin the article’s worked examples, such as Recall@5 = 0.750, MRR = 0.625, and nDCG@5 = 0.627, so you can check each metric against a hand calculation.

  Run it: make eval runs the suite. make benchmark calls hosted GPT, Claude, and Gemini models and needs their API keys.

  Repository · Article

• TypeScript Agent Service

  A pnpm monorepo for Python ML engineers who need to ship TypeScript: two agent-loop implementations, a streaming API, Postgres run history, a worker, and an MCP server.

  Result: 40 tests, of which 36 run without Postgres or an API key. The agent-loop tests finish in under a second with no network.

  Run it: Node 24 and pnpm. pnpm demo needs no API key, database, or Docker. The real API needs OPENAI_API_KEY and Postgres in Docker.

  Repository · Article

• Search Ranking Stack

  BM25, dense retrieval, hybrid fusion, cross-encoder reranking, and LLM reranking on a sample of Amazon ESCI product-search data, measured stage by stage.

  Result: NDCG@10 is 0.585 for BM25, 0.611 for dense retrieval, and 0.628 for hybrid fusion, which beats either method alone. Recall@100 is fixed at retrieval (0.842 for hybrid), and later stages cannot raise it.

  Run it: uv sync, then uv run download-data (2.5 GB download, 5 MB sample). The LLM stage runs through Ollama, the Anthropic API, or a local Qwen model.

  Repository · Article

• Model Compression Demo

  A CLI that plans LLM quantization for a given model and target hardware, then prints recipes, dry runs, serving commands, and benchmark plans for vLLM, LLM Compressor, and TensorRT-LLM.

  Result: No measured result. The benchmark numbers in the article come from other teams; the demo gives you the commands to measure your own.

  Run it: uv run python demo.py plan and the other planning commands need no GPU. Quantization, serving, and quality runs need a GPU and vLLM.

  Repository · Article