Prompt, RAG, fine-tune: three ways to shape a model

This is post 8 of 8 in the Foundations series. The first seven posts built up to one practical question: when you have an LLM and a use case, how do you make it work for you?

There are three levers. They aren't substitutes; they fix different things, at different costs.

The three levers

Prompting. Change what you write in the message. System prompts, instructions, examples in-context, output format hints. No code, no infrastructure. Iteration is seconds.

RAG. Retrieve relevant data from your own corpus and paste it into the context window before the question. Some infrastructure (a vector store, an embedding model, retrieval code), but no model training. Iteration is minutes to hours.

Fine-tuning. Continue training the model on your own examples so new behaviours get baked into the weights. Real infrastructure (training pipeline, GPU budget, eval harness), and re-training every time you want to change. Iteration is hours to days, sometimes more.

All three change what comes out of the model. None of them change what the model is, except fine-tuning, which changes a copy of it.

What each one actually changes

Prompting changes the input. The weights are frozen. The context window is full of whatever you wrote. Good prompting is mostly about giving the model the right framing, the right format, and one or two concrete examples ("few-shot"). The biggest under-rated technique is just telling the model the role it should play and the structure of the output you want. Most "the model isn't doing what I want" problems are solved here.

RAG changes the input too — it just generates the input dynamically per query, based on retrieved data. The weights are still frozen. The model sees a different prompt every time, depending on what was relevant. RAG fixes "the model doesn't know my data". It does not fix "the model talks in the wrong tone" — that's still a prompting problem.

Fine-tuning changes the weights. The numbers in the model file are different after fine-tuning than before. Patterns that were rare in the original training become common in the fine-tuned version. Used right, fine-tuning bakes in behaviours and styles. Used wrong, it makes the model worse at general tasks (catastrophic forgetting) without making it meaningfully better at the target task.

What each one costs

Numbers that hold roughly true in 2026:

• Prompting: free in dollars. Costs your time. Most of the cost is iteration cycles, which is why short-cycle prompt-and-test loops matter more than clever prompts.
• RAG: tens to hundreds of dollars to build a small system, low monthly bill for vector storage, plus your inference bill. Embedding 10 million tokens costs roughly a dollar at current OpenAI rates.
• Fine-tuning: cheapest case is a hosted fine-tune of a small model, a few hundred dollars. Largest case — a custom fine-tune of a 70B+ model — is tens of thousands. Plus the eval cost; without good evals, fine-tuning is a coin flip.

All three add inference cost on top, since you're still calling the model once per request. RAG adds the most, because it grows the prompt size.

When to pick which

My rules of thumb, after building real things with all three:

Start with prompting. Always. Most problems are prompting problems. If you can fix it in the system prompt, do that and stop.

Reach for RAG when the model needs facts it doesn't have. Your docs, your codebase, your customer's account history, anything that changes or is private. RAG is the right answer 80% of the time when prompting alone isn't.

Reach for fine-tuning when you've maxed prompting and RAG, and the gap is style or behaviour. Specific output formats, domain-specific tone, classification tasks at scale where the prompt would otherwise be huge. Fine-tuning is rarely the first answer; it's often the last 10%.

A mistake I see often: teams jumping to fine-tuning because it sounds technical, then discovering they were really fixing a prompting problem. The fine-tune works, but it would have worked as a prompt for a fraction of the cost.

Another mistake: stuffing everything into the prompt as the system grows. Past a certain size, the prompt is its own corpus, and the right move is to retrieve from it instead of always sending the whole thing.

Combine them

The real systems run all three at once. A production setup looks like:

• A carefully written system prompt with role, format, and constraints.
• RAG fetching the relevant slice of your corpus per query.
• Optionally, a fine-tuned model that's already good at your specific output format.

Each lever is multiplicative. A great prompt with the wrong context still fails. A great context with a sloppy prompt still fails. A great fine-tune with neither doesn't matter.

What to take away

• Three levers, three different problems: prompting (how you ask), RAG (what data the model can see), fine-tuning (what behaviour the model defaults to).
• Costs scale up the same way: prompting is free, RAG is cheap, fine-tuning is expensive and only sometimes worth it.
• Default order: prompt first, then RAG, then fine-tune. Most teams do this in reverse and burn money.
• In production, all three combine. None of them is the answer alone.

That closes the Foundations series. Eight posts, one arc: from "AI is a marketing word" to "here's exactly which lever to reach for and why". The next series picks up from a different angle — now that you understand these models, where do they actually run? Cloud, your laptop, the device in your hand. Tradeoffs, hardware, runtimes.