Fine-tuning vs RAG vs prompting: which should you use?
Use prompting to clarify the task, retrieval-augmented generation (RAG) to supply current or private knowledge, and fine-tuning to change repeatable model behavior. These techniques solve different problems and often belong in the same system.
Start with a measured prompt baseline. Add RAG when answers need external evidence or frequently changing facts. Consider fine-tuning for persistent errors in tone, classification, tool use, or domain language when enough reviewed examples exist. For response-format errors, try schema-constrained output first.
Choose from observed errors, how often the facts change, and the data available to test a fix.
Decision table
| Problem | First intervention | Why |
|---|---|---|
| Instructions are unclear or examples are missing | Prompting | It tests whether clearer instructions and examples solve the observed failures. |
| Facts are private, cited, or change frequently | RAG | Retrieval updates knowledge without retraining and can preserve source provenance. |
| Stable behavior still fails after prompt testing | Fine-tuning candidate | Reviewed examples can teach style, labels, or tool behavior; compare against the prompt baseline. |
| A careful prompt baseline still fails | Test another model or system design | Compare alternatives on the same cases; prompt failure alone does not show whether tuning would help. |
| Answers need current facts and specialized behavior | RAG plus fine-tuning | Retrieval supplies evidence while tuning changes how the model uses it. |
Use prompting first
Prompting is a low-setup way to diagnose failures. Define the task, output contract, examples, allowed tools, and refusal conditions. Build a small evaluation set before adding more infrastructure. If a stronger prompt fixes the representative failures, stop there.
If a growing prompt becomes harder to maintain, identify what was added. Changing product facts may belong in retrieval. Repeated behavior examples may be training candidates. Prompt length alone does not prove that either change will improve the result.
Use RAG for knowledge and provenance
RAG fits product manuals, policies, internal documents, recent events, and answers that must cite a source. The original RAG paper combines a generator with retrieved passages; the retrieved evidence is supplied at request time. Its quality depends on parsing, chunking, retrieval, ranking, permissions, and citation support. Changing the generator alone does not supply a source that retrieval omitted.
RAG is not a behavior-training method. It can show the model an example or rule, but it does not make that behavior stable across every request.
For a support assistant, an outdated warranty answer points to missing current policy: supply that policy through retrieval. If the correct policy is already present but the assistant repeatedly assigns the wrong complaint category, test clearer examples first. Fine-tuning becomes a candidate if that stable classification error remains and reviewed training examples exist.
Use fine-tuning for repeatable behavior
Supervised fine-tuning trains on input/output examples, as described in the TRL training guide. It is a candidate for classification, extraction style, response structure, domain terminology, and recurring tool-use patterns. For format-only failures, first try schema-constrained output if your runtime supports it. A valid structure still needs checks for correct field values.
Fine-tuning requires reviewed training data, a held-out evaluation set, model versioning, and a rollback path. Do not fine-tune facts that change often when retrieval can provide them at request time.
A practical sequence
- Define success and build a representative evaluation set.
- Establish a prompt-only baseline with the strongest acceptable model.
- Add RAG if failures come from missing or changing evidence.
- Collect and review examples of the stable failures that remain.
- Fine-tune only if the quality, latency, or cost gain beats the extra data and model operations.
- Re-run the same evaluation after every prompt, index, model, or adapter change.
Deeper reading
- Fine-Tuning LLMs covers data, LoRA, evaluation, and deployment.
- RAG Evaluation Metrics shows how to locate failures by pipeline stage.
- Context Engineering for AI Agents covers the runtime context around prompts and retrieval.