Prompt, RAG, fine-tune: three ways to shape a model

This is post 8 of 8 in the Foundations series. The first seven posts were all building toward one practical question: you've got an LLM and a use case, so how do you actually make it work for you?

There are three levers. They aren't substitutes. Each one fixes a different problem, at a different cost.

The three levers

Prompting. Change what you write in the message: system prompts, instructions, examples in-context, hints about the output format. No code, no infrastructure. You iterate in seconds.

RAG. Pull relevant data from your own corpus and paste it into the context window before the question. This needs a bit of infrastructure (a vector store, an embedding model, some retrieval code) but no model training. You iterate in minutes to hours.

Fine-tuning. Keep training the model on your own examples so new behaviour gets baked into the weights. This needs real infrastructure (a training pipeline, a GPU budget, an eval harness) and a fresh re-training run every time you want a change. You iterate in hours to days, sometimes longer.

All three change what comes out of the model. None of them change what the model is. The one exception is fine-tuning, and even then it only changes a copy.

What each one actually changes

Prompting changes the input. The weights stay frozen and the context window holds whatever you wrote. Good prompting is mostly about handing the model the right framing, the right format, and a concrete example or two ("few-shot"). The most under-rated trick is simply telling the model what role to play and what shape the output should take. Most "the model isn't doing what I want" problems die right here.

RAG also changes the input. It just builds that input fresh for every query out of whatever it retrieved. The weights are still frozen, and the model sees a different prompt each time depending on what was relevant. RAG fixes "the model doesn't know my data." It does nothing for "the model talks in the wrong tone." That one is still a prompting job.

Fine-tuning changes the weights. The numbers in the model file are literally different afterwards. Patterns that were rare in the original training become the default in the fine-tuned version. Done well, it bakes in behaviours and styles. Done badly, it makes the model worse at general tasks (catastrophic forgetting) without making it meaningfully better at the one you cared about.

What each one costs

Figures that hold roughly true in 2026.

• Prompting is free in dollars. It costs your time, and most of that time is iteration cycles, which is why a fast prompt-and-test loop beats clever prompts.
• RAG runs tens to hundreds of dollars to stand up a small system, a low monthly bill for vector storage, plus your inference cost. Embedding 10 million tokens is about a dollar at current OpenAI rates.
• Fine-tuning at its cheapest is a hosted fine-tune of a small model for a few hundred dollars. At its largest, a custom fine-tune of a 70B-plus model, it runs into tens of thousands. Add the eval cost on top, because without good evals a fine-tune is a coin flip.

All three pile inference cost on top, since you're still calling the model once per request. RAG adds the most, because it grows the prompt.

When to reach for which

These are my rules of thumb, after building real things with all three.

Start with prompting. Always. Most problems are prompting problems in disguise. If the system prompt can fix it, fix it there and stop.

Reach for RAG when the model needs facts it doesn't have. Your docs, your codebase, a customer's account history, anything private or anything that changes. When prompting alone isn't enough, RAG is the right answer roughly 80% of the time.

Reach for fine-tuning only once you've maxed out prompting and RAG and the gap that's left is style or behaviour. Fixed output formats, a domain-specific tone, classification at a scale where the prompt would otherwise be enormous. Fine-tuning is rarely the first answer. It's usually the last 10%.

I see the same two mistakes over and over. The first: teams jump straight to fine-tuning because it sounds impressively technical, then realise they were fixing a prompting problem all along. The fine-tune works, sure, but a prompt would have worked for a fraction of the cost. The second: cramming everything into the prompt as the system grows, until the prompt is its own corpus and the right move is to retrieve from it instead of shipping the whole thing every time.

Combine them

Real systems run all three at once. A production setup tends to look like this.

• A carefully written system prompt carrying the role, the format, and the constraints.
• RAG fetching the relevant slice of your corpus for each query.
• Optionally, a fine-tuned model that's already fluent in your specific output format.

The levers multiply, they don't add. A great prompt with the wrong context still fails. Great context with a sloppy prompt still fails. A great fine-tune with neither barely matters.

And that closes the Foundations series. Eight posts, one arc: from "AI is mostly a marketing word" to "here's the exact lever to reach for, and why."

You now know what these models are and how to bend them to your needs. The obvious next question is where they actually run. The cloud? The laptop on your desk? The phone in your pocket? Each choice carries its own tradeoffs in hardware, cost, privacy, and speed, and that's the whole subject of the next series, AI Running.