Somewhere in your company right now, someone is proposing to fine-tune a model. I’d put money on it. Fine-tuning sounds serious, sounds proprietary, sounds like a moat. And nine times out of ten, it’s the wrong first move.
I say this as someone who is actively planning a fine-tune — a small local model trained on my own writing. But I’m doing it after exhausting the cheaper levers, and knowing exactly why the cheaper levers can’t do this particular job. That ordering is the entire skill.
I’m not an ML engineer. I spent 10 years in startup operations before building Dszape, a hotel management SaaS, solo with AI. So this is the plain-language version of the fine-tuning vs RAG vs prompting decision — the one I’d give a founder over coffee, not the one from a research paper.
The Three Levers, in Plain Language
You have three ways to make an AI model better at your specific problem. They differ in what they change.
Prompting changes what you ask. You write better instructions: context, examples, rules, the format you want back. The model itself is untouched. Think of it as briefing a very capable freelancer really well.
RAG changes what the model can see. Retrieval-Augmented Generation is a fancy name for a simple move: before the model answers, your system fetches the relevant documents (from your docs, database, knowledge base) and pastes them into the prompt. The model isn’t smarter; it’s informed. Think of it as handing the freelancer the correct file before they start.
Fine-tuning changes what the model is. You take an existing model and continue training it on your own examples until its default behaviour shifts. Think of it as hiring the freelancer full-time and having them absorb your house style for six months.
The costs scale the same way. Prompting costs you an afternoon. RAG costs you a pipeline you must build and maintain. Fine-tuning costs you a dataset, training runs, evaluation, and re-doing all of it whenever things change.
What Each Lever Is Actually For
Here’s the part that gets muddled in most explanations. The three levers solve three different problems, and the biggest mistakes come from using one to solve another’s problem.
Prompting solves behaviour problems. Wrong tone, wrong format, ignoring instructions, inconsistent output. If you can describe what “good” looks like, you can usually prompt your way there. Modern frontier models follow long, detailed instructions astonishingly well — my entire development workflow on Dszape runs on carefully engineered instructions and project rule files, no fine-tuning anywhere.
RAG solves knowledge problems. The model doesn’t know your refund policy, your product catalogue, your customer’s history — because it can’t. That information is yours, and it changes weekly. No amount of clever prompting can make a model know facts it was never given, and fine-tuning is a terrible way to teach facts (more below). If the problem is “the model doesn’t know X”, the answer is almost always retrieval.
Fine-tuning solves style and reflex problems. You want the model’s default behaviour changed: a specific voice, a strict output format followed thousands of times a day, a narrow classification task done fast and cheap by a small model. Fine-tuning bakes behaviour in so you stop paying for it in prompt tokens on every single call.
A one-line test I use: is the gap in the instructions, the information, or the instincts? Instructions → prompt. Information → RAG. Instincts → fine-tune.
The Decision Tree I Actually Use
Run your problem through this, in order. Do not skip steps — the order is the point.
Step 1: Have you genuinely exhausted prompting? Not “we tried a prompt”. Detailed instructions, worked examples, explicit rules, output templates, splitting one big task into smaller chained ones. In my experience most “the model can’t do this” complaints die at this step. Prompting is free, instant, and reversible; you should be embarrassed to spend money before this lever is fully pulled.
Step 2: Is the failure caused by missing information? If the model gets facts wrong or doesn’t know your data — that’s retrieval. Start with the dumbest version that works: paste the relevant document into the prompt by hand. If that fixes it, you’ve proven the diagnosis, and now you’re just automating the paste. That’s all RAG is.
Step 3: Is the remaining gap about style, format, or unit cost at volume? Only now does fine-tuning enter the conversation. And even then, ask: is this worth a dataset of hundreds-to-thousands of quality examples, training runs, an evaluation harness, and retraining whenever things drift? Sometimes yes. Usually no.
Step 4: Most real systems end up hybrid. Prompting for behaviour, RAG for knowledge, and — occasionally — a small fine-tuned model for one narrow, high-volume task. These are layers, not competitors.
Why Most Startups Should Stop at Steps 1 and 2
Let me make the case bluntly, operator to operator.
Fine-tuning is a commitment, not a feature. The moment you fine-tune, you own a model lifecycle: dataset curation, training, evaluation, regression testing, retraining. That’s a standing operational cost in your scarcest currency — attention. I’ve watched startups burn a quarter on a fine-tune that a week of prompt engineering would have matched.
Fine-tuning freezes; your business moves. Bake this month’s knowledge into weights and next month it’s stale, and the fix is another training run. RAG handles change by design — update the document, and the very next query sees it.
Fine-tuning is a bad way to teach facts. This is counterintuitive but well-established: models fine-tuned on factual material don’t reliably recall it; they mostly pick up the style of it. If you need the model to cite your actual policy correctly, put the policy in the context window (RAG). Weights are for instincts, context is for facts.
The frontier keeps eating fine-tunes. The strongest argument of all: base models improve fast. A fine-tune that beat last year’s model often loses to this year’s base model with a good prompt. Every rupee spent on prompting and retrieval carries forward to the next model generation; a fine-tune may not.
This is the same discipline as any build-vs-buy call, and my bias is the same one I wrote about in Build vs Buy: When to Use AI in Your Startup: take the cheap, reversible option until it demonstrably fails.
For scale: Dszape is a production platform — 245+ API routes, 719+ source files, a booking and billing engine — built almost entirely with AI, and there is not one fine-tuned model in it. Prompting plus project rules plus feeding the model the right context has carried the whole thing. (How that works day to day is in my solo building journey.)
When Fine-Tuning Is Right: My Voice Model Experiment
So why am I — Mr Exhaust-the-Cheap-Levers — planning a fine-tune?
The experiment: take a small open model and fine-tune it on my own writing — years of it — to get a “voice model” that drafts the way I actually write, running locally on my MacBook.
Walk it through the tree and you’ll see why it passes:
- Prompting? Tried extensively. “Write like this, here are five samples” gets you a decent imitation that drifts within a few paragraphs. Voice is a thousand small reflexes — rhythm, word choice, where a sentence breaks. It’s instincts, not instructions, which is exactly the gap prompting can’t close.
- RAG? Wrong tool by definition. Retrieval supplies facts; my problem isn’t missing facts, it’s default behaviour.
- Stakes? Low. It drafts; I edit. A bad output costs me nothing. (Compare that with production code, where I use the frontier — the reasoning is in Local LLM vs Cloud API.)
- Volume and privacy? High and personal. It’s my private writing as training data, and I’ll iterate constantly — which makes a small local model on my 36 GB MacBook the natural home. The hardware side of that is covered in my local LLM guide.
Style problem, low stakes, private data, high iteration, small model. That’s the fine-tuning bullseye. If your project doesn’t look roughly like that, be suspicious of your own enthusiasm.
Notice also what the voice model is not: it’s not how I give my AI memory of me. Voice is instincts, so it goes in the weights. Facts about my life and work go in retrievable files — a completely different mechanism, which I’ve written up in Giving AI Agents Long-Term Memory.
Quick Reference: Symptoms and Prescriptions
- “The output format/tone is wrong” → Prompting.
- “It ignores some of my instructions” → Prompting (shorter, structured, chained tasks).
- “It doesn’t know our data/policies/catalogue” → RAG.
- “It makes up facts about our domain” → RAG (ground it in real documents).
- “Answers must reflect data that changes weekly” → RAG. Never fine-tuning.
- “We need a specific voice/persona at scale” → Fine-tune, after prompting proves insufficient.
- “One narrow task, huge volume, unit cost matters” → Fine-tune a small model.
- “We want a moat” → Not a symptom. Go back to step 1 and find the actual problem.
FAQ
What’s the difference between RAG and fine-tuning, in one sentence?
RAG puts your information in front of the model at question time; fine-tuning changes the model’s default behaviour by training on your examples. Facts belong in retrieval, instincts belong in weights.
How much data do I need to fine-tune a model?
For style and format tasks using efficient methods (LoRA on a small model), hundreds to a few thousand high-quality examples can be enough. Quality dominates quantity — a small clean dataset beats a large messy one. If you can’t assemble a few hundred good examples, you’re not ready to fine-tune.
Is fine-tuning expensive in 2026?
The compute is cheaper than people assume — small-model training runs cost tens of dollars, not thousands. The real cost is everything around it: curating the dataset, building evaluation, and retraining as things drift. Budget for the lifecycle, not the training run.
Can fine-tuning make a model know my company’s information?
Not reliably, and this is the most common expensive mistake. Fine-tuned models absorb the style of training material far better than the facts in it. For factual accuracy about your business, use RAG so the real document is in the context window when the model answers.
Should I fine-tune to make the model follow my instructions better?
Try a frontier model with better-structured prompts first. Instruction-following in current top models is excellent, and most “it won’t listen” problems are prompt problems. Fine-tune for instruction-following only on small models where you’re optimising cost at high volume.
I write about building real software solo with AI — what I actually use, what I skip, and what breaks in production. If this guide saved you a fine-tuning project, consider it my good deed for the week. More at my complete solo SaaS journey, or read more about me here.
