RAG vs. fine-tuning vs. prompt engineering: when each is correct

"Should we use RAG, fine-tune the model, or just write a better prompt?" is the most common question we get from teams past the demo stage, and the honest framing is that RAG vs. fine-tuning vs. prompt engineering isn't a single choice — it's three separate questions, and most teams skip straight to the expensive one. Default to prompt engineering first because it is the cheapest and fastest to test. Reach for RAG the moment the correct answer depends on information that changes after training day. Fine-tune only when the model needs to reliably use a specific vocabulary, format, or reasoning pattern that no amount of prompting produces consistently — and even then, the research says pair it with retrieval rather than picking one over the other.

Anthropic's own prompt engineering guidance is explicit about the ordering: try a clearer prompt before reaching for fine-tuning, because prompting is free to iterate on and testable in seconds, while fine-tuning means preparing a training set, running and evaluating a training job, and living with a model that's now harder to update than a prompt would have been (Anthropic, Prompt engineering best practices for 2026, 2026). That's not a preference for the cheaper option out of caution — it's the same test any engineer already runs before reaching for a bigger tool: does a clearer instruction, a worked example, or an assigned role get the model to a consistently correct answer, before spending days on a training run to teach it something a paragraph in the prompt might have taught it in one API call. Most of the "we need to fine-tune" conversations we have start here, and a surprising number end here too — once the prompt actually states the format, the edge cases, and one or two worked examples explicitly, the gap that looked like it needed retraining turns out to have been an underspecified instruction.

RAG earns its complexity exactly where prompting runs out: when the correct answer depends on something outside the model's training data, and — critically — when that something changes. A support policy that gets updated monthly, a product catalog that changes daily, a client's own documents that didn't exist when the model was trained: none of that can live in a prompt or a fine-tune, because both are fixed at write time, and re-training every time a policy changes is not a serious operating model for anyone. Retrieval also buys something fine-tuning structurally can't: a citation. A RAG system can point to the exact paragraph an answer came from; a fine-tuned model has folded that knowledge into its weights and can't show its work, which matters the moment a compliance officer, a lawyer, or an auditor asks where an answer came from — the same distinction we've written about at length in RAG is a data problem wearing an AI costume, where the actual failure mode is almost never model choice, it's the retrieval pipeline underneath it.

Fine-tuning earns its cost where RAG structurally can't help: teaching a model a specific vocabulary, tone, or output format it needs to use consistently, or cutting the latency and infrastructure of a retrieval call out of the response path entirely for a domain that doesn't change often. And the research says the two aren't actually a fork in the road — they compound. Microsoft Research's own study of fine-tuning and RAG pipelines across GPT-3.5, GPT-4, and Llama2-13B, tested on an agricultural question-answering task, found fine-tuning alone lifted accuracy by more than 6 percentage points over the base model, and layering RAG on top of the fine-tuned model added another 5 points on top of that — with one specific test showing answer similarity climb from 47% to 72% once the fine-tuned model could also draw on information from outside its own training data (Microsoft Research, RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture, 2024). The two techniques aren't competing for the same job — fine-tuning teaches the model how to talk about a domain, retrieval tells it what's true in that domain today, and a system that only does one is leaving the other's gain on the table.

We've built a version of that combination directly: Vaisela's AI seller assistant for Amazon runs a custom-trained LLaMA model — not a generic API call — because the recommendations it makes depend on reasoning patterns and terminology specific to Amazon Seller Central that a prompt alone couldn't hold consistently across thousands of daily recommendations. But the model isn't left to run on its training data alone: it works from an explicit knowledge hierarchy that prioritizes Amazon's own terms, academy training material, and mentorship content over anything else it might otherwise reach for — retrieval-style grounding layered on top of the fine-tuned base, the same combination the research above found compounds rather than competes. Neither piece was optional: a prompted-only model wouldn't have held the domain vocabulary consistently, and a retrieval-only model would have had no consistent reasoning pattern to apply to what it retrieved.

In practice the decision comes down to three questions, asked in order, not all at once. First: can a clearer prompt with explicit examples get a consistently correct answer today? If yes, ship that and stop — it's the fastest path and the easiest to change tomorrow. Second: does the correct answer depend on something that changes after training day, or does the answer need a citation someone can check? If yes, that's RAG, and the retrieval pipeline — not the model — is where the engineering effort actually belongs. Third: does the model need to reliably use a vocabulary, format, or reasoning pattern that prompting can't hold consistently, and is the domain stable enough that periodic retraining is worth it? If yes, that's fine-tuning, and per the research above, it's worth checking whether pairing it with retrieval gets more than either would alone. Getting this order backwards is the expensive mistake: teams that fine-tune first because it sounds like the "real" solution end up paying for a training run to solve a problem a better prompt would have fixed in an afternoon, or discover months later that their fine-tuned model still can't cite where an answer came from — because that was never a fine-tuning problem to begin with.

If you're mid-decision on this right now, the fastest way to get it right is to test in the order above rather than starting with whichever technique sounds most sophisticated: a week spent tightening a prompt costs a fraction of a week spent fine-tuning the wrong thing. That's the same sequencing our generative AI development services team runs on every engagement — prompt and retrieval before training, and training only where the research and the domain both say it'll hold. If retrieval is already the right call and what's missing is the pipeline's own tuning — chunking, reranking, evaluation against a golden set — the fix is a narrower, specific hire: hire RAG engineers who've tuned retrieval for production before, not built a demo once.