Production RAG Pipeline: Which Stages Should I Evaluate?
Evaluate document preparation, evidence retrieval, and answer generation separately. A production RAG pipeline must preserve usable source content, retrieve permitted evidence, and produce an answer supported by that evidence. The final answer alone cannot identify which stage failed.
Begin with a question whose correct source passage is known.
Check the evidence before changing the model
| Stage | What to retain | Check |
|---|---|---|
| Parse and chunk | Source revision, page or section, chunk identity | Required text and tables survive preparation |
| Embed and index | Encoder identity, dimensions, metadata, permissions | Queries use compatible representations and access filters |
| Retrieve | Candidate identities, scores, applied filters | Expected permitted evidence is present |
| Rerank and assemble | Kept and removed candidates, actual model input | Relevant facts remain with enough surrounding context |
| Generate | Answer and cited sources | Each consequential claim has support |
Consider a question asking for a warranty exclusion from a PDF. If parsing omitted the exclusion table, changing the generator will not restore it. If the correct chunk was retrieved but removed during reranking, repair candidate selection. If it reached the prompt but the answer contradicts it, investigate evidence use and generation.
Add retrieval stages for measured misses
Dense retrieval helps with paraphrases. Sparse retrieval helps with exact terms and identifiers. Qdrant’s hybrid-query documentation describes combining retrieval results through methods such as reciprocal rank fusion. Whether hybrid retrieval improves your task still requires relevance labels.
Reranking can improve candidate order but adds computation. Test it on the retrieved candidates, with latency included. Also inspect where relevant passages appear in the generator’s context: Lost in the Middle found position-dependent performance in its tested long-context tasks.
Apply permissions before unauthorized source content reaches the model, and carry source identity through to citation checks. Test updates and deletions as well as new ingestion.
For choosing basic, agentic, or graph retrieval, see RAG architecture. The production-pipeline section of the LLM Engineering Guide explains the stage relationships.