"Should we use RAG or fine-tune the model?" is the most-asked question in AI project kick-offs, and it's usually the wrong question. They solve different problems. Confusing them costs 2–6 months and a mid-six-figure budget.
TL;DR: RAG injects knowledge. Fine-tuning teaches behaviour. If the problem is "the model doesn't know our data", use RAG. If the problem is "the model's outputs don't match our style, format, or domain reasoning patterns", consider fine-tuning. Most production systems end up hybrid.
What RAG is actually for
Retrieval-Augmented Generation lets an LLM answer questions about data it was not trained on. You index your docs, retrieve the relevant chunks at query time, inject them into the prompt, and the model answers grounded in those chunks. Strengths:
- Knowledge is fresh — update the index, the model sees new data instantly.
- Attribution is natural — cite the retrieved chunks.
- Governance is tractable — change what's indexed, change what the model can say.
- Cheap to iterate — no training costs.
Weaknesses: can't teach behaviour, output style, or reasoning patterns. If the base model reasons poorly in your domain, RAG won't fix it.
What fine-tuning is actually for
Fine-tuning modifies the weights of a smaller model on your data so it behaves a certain way — speaks your style, follows your format, reasons in your domain. Strengths:
- Output style is consistent — brand voice, legal disclaimers, specific format.
- Domain-specific reasoning — medical, legal, financial, where generic models over-hedge or under-cite.
- Smaller, cheaper models can be taught to behave like larger ones for a narrow task.
- Latency — a fine-tuned 7B model is much faster than GPT-4 class.
Weaknesses: expensive (training + compute + eval), stale (knowledge is baked in), harder governance (can't "unlearn" quickly).
The decision tree
- Is the problem "model doesn't know our data"? → RAG.
- Is the problem "model's output format is wrong"? → Prompt engineering first. If that fails, fine-tune.
- Is the problem "model's reasoning is wrong in our domain"? → Fine-tune.
- Is the problem "we need cheap low-latency inference at scale"? → Fine-tune a smaller model.
- Is it several of the above? → Hybrid — RAG on top of a fine-tuned model.
Cost comparison (2026, realistic)
| Dimension | RAG | Fine-tuning |
|---|---|---|
| Initial engineering | £20k–£60k | £40k–£150k |
| Training compute | £0 | £5k–£100k |
| Per-query cost | Higher (large-context prompts) | Lower (smaller model) |
| Update cycle | Instant (reindex) | Weeks (retrain) |
| Governance | Edit the index | Retrain |
When hybrid is the right call
You fine-tune a smaller model to talk in your voice and follow your format (e.g., Llama 3 8B for customer support), then use RAG on top to inject fresh ticket content. Best of both: consistent behaviour, fresh knowledge. Downside: you now maintain two systems.
Our rule of thumb: start with RAG on GPT-4 / Claude Opus / Gemini Pro. If per-query cost is killing you or output quality isn't consistent, graduate to fine-tuned small model + RAG.
Common mistakes
- Fine-tuning when prompt engineering would have worked. Try 3 iterations of prompt + few-shot before assuming you need fine-tuning.
- RAG without an eval harness. You will not know it's broken until users complain.
- Fine-tuning on 200 examples. Too little data; results are unstable. Usually need 2,000–20,000 well-curated examples.
- Choosing one over the other on vibes. Write the decision tree out; show it to a skeptic.
Governance considerations
RAG has a cleaner compliance story for regulated domains — you can demonstrably show what the model retrieved. Fine-tuned models are harder to audit because knowledge is baked into weights. For GDPR / HIPAA / legal / medical, most teams default to RAG for this reason alone.
FAQ
Can I just fine-tune and skip RAG entirely?
Only if your data is small, stable, and the model just needs to learn a behaviour. Most real systems need both fresh data and consistent output.
How do I evaluate whether RAG or fine-tuning is working?
Automated eval harness with retrieval recall@k, response accuracy against a ground-truth set, and human spot-checks. Without eval you are guessing.
Is fine-tuning dead now that GPT-4 exists?
No — it's more targeted than before. Fine-tuning makes sense for latency-sensitive, high-volume narrow tasks where a 7B or 13B model with your behaviours beats paying GPT-4 per query.
Next step
Tell us your problem and we'll recommend RAG, fine-tuning or hybrid in a 30-minute call. Read about our AI development services, hire LangChain engineers, hire OpenAI integration engineers, or hire Hugging Face specialists.