RAG vs Fine-tuning
two ways to "teach" an LLM, and picking the wrong one costs you
AI engineeringAI engineering
Almost everyone reaches for fine-tuning first when the model "doesn't know our stuff." Usually that's the wrong tool. Here's the split intermediate engineers actually need.
The core distinction
Both make an LLM more useful for YOUR domain — but they change different things:
- RAG (Retrieval-Augmented Generation) → changes what the model KNOWS at answer time. You fetch relevant docs and inject them into the prompt. Knowledge lives OUTSIDE the weights.
- Fine-tuning → changes how the model BEHAVES. You keep training on example pairs so patterns get baked INTO the weights. Behavior lives INSIDE the weights.
One-liner: RAG for knowledge, fine-tuning for form.
Side by side
RAG Fine-tuningBest for facts, docs, "what" style, format, "how"
Update a fact re-index a file (secs) re-train (hours + $$)
Data needed your documents 100s–1000s of examples
Failure mode stale / missing chunk confident wrong tone
Citations easy (you have source) hard
Cost per change cheap expensive
Concrete example: "Answer support questions about our API."
- The API changed yesterday → RAG. Re-index the new docs and the model reads them fresh. A model fine-tuned last week is already stale.
- Every reply must follow your exact 3-part support format and tone → fine-tuning. No amount of retrieved text reliably enforces output SHAPE.
The strongest real systems use BOTH: fine-tune the format, RAG the facts.
The gotcha that trips people
"The model gives wrong facts, so let's fine-tune on the correct facts." Fine-tuning teaches patterns, not a reliable fact store — the model will still blend or forget specifics, and now you can't cite a source or fix it without retraining. Wrong facts → almost always a RAG/retrieval problem, not a training one.
Second misconception: fine-tuning does NOT cut token cost by "memorizing your docs." It changes behavior, not your context budget.
Your exercise
Take one task you'd want an LLM to do for your work. Answer two questions:
(a) Is the hard part missing KNOWLEDGE or wrong BEHAVIOR/format?
(b) How often does the underlying info change?
Your answers should point clearly at RAG, fine-tuning, or both. Which did you land on — and why?