RAG: giving a model memory it doesn't have

Post 6 ended with: an LLM only knows what's in its weights and what's in its context window right now. RAG is the most common way of getting the right things into that window.

This is post 7 of 8 in the Foundations series.

What RAG actually means

RAG stands for Retrieval-Augmented Generation. It's three steps:

1. Retrieve. Take the user's question. Search a corpus (your docs, your codebase, your wiki) for the most relevant chunks of text.
2. Augment. Stick those chunks into the LLM's prompt, alongside the question.
3. Generate. Let the LLM answer, with the fetched text now visible in its context window.

That's it. There's no special model required. RAG works with the same LLM you'd use without it. The only thing that changes is what's sitting in the context window when the model answers.

The word "retrieval" makes it sound exotic. It isn't. It's a search query whose results get pasted into the prompt before the question. Every RAG system, from a 100-line script to a full enterprise stack, is a variation on that pattern.

How retrieval is actually done

The naive version of step 1 is keyword search. It works for some cases. The standard 2026 version is vector search using embeddings.

An embedding is a list of numbers that represents the meaning of a piece of text. Two pieces of text with similar meanings have embeddings that are close together in space, even if they don't share keywords. "How do I cancel my subscription?" and "Stop my membership" embed near each other; a keyword search would miss the second one.

The pipeline:

• Once, ahead of time: split your corpus into chunks (paragraphs, sections), run each chunk through an embedding model, store the resulting vectors in a vector database (Pinecone, Weaviate, pgvector, or just a flat file for small corpora).
• At query time: embed the user's question, find the K nearest chunks in the vector store, return those chunks.

Real systems usually do hybrid retrieval — vector search plus keyword search, with a re-ranker on top. Pure vector search has its own failure modes (it'll happily return semantically related chunks that don't actually answer the question). Production RAG is mostly the engineering of getting retrieval to be both relevant and precise.

Why RAG isn't fine-tuning

This distinction trips a lot of people up.

Fine-tuning changes the model's weights. You take an existing LLM and continue training it on your data, so the new patterns get baked into the parameters. Expensive, slow, irreversible without rolling back to the old weights.

RAG doesn't touch the model. The weights are frozen. The model is, every single query, given fresh text in its context. The model has no memory of the previous query. If you update your docs, RAG sees the updated version on the next query. Fine-tuning would need a re-train.

The practical implication: for changing factual knowledge, RAG is almost always the right tool. Knowledge changes. Re-training is expensive. Updating an index is cheap.

For changing the model's style, behaviour, or domain-specific reasoning, fine-tuning is sometimes the right tool. Knowledge that's stable, reasoning patterns specific to your data, output formats — those bake well.

When RAG fails

RAG isn't a silver bullet. Three common failure modes:

The retrieval misses. If your search returns the wrong chunks, the LLM has nothing relevant to ground in. It'll either say "I don't know" (if you're lucky and prompted well) or hallucinate (if you weren't). Most "my RAG system is bad" complaints are retrieval problems, not LLM problems.

The chunks are too small or too big. Tiny chunks lose context. Huge chunks crowd out other relevant material in the window. There's no universal right size — between 200 and 1000 tokens is typical, tuned per corpus.

The model ignores the context. This is the lost-in-the-middle problem from post 6. If you stuff 50 chunks into the prompt and the relevant one is at chunk 25, the model might miss it. Re-rankers and ordering tricks help.

A RAG system is only as good as its weakest link, and the weak link is almost always retrieval.

What to take away

• RAG is: search your corpus, paste relevant chunks into the prompt, ask the LLM. The whole pattern is at the prompt layer; the model itself doesn't change.
• Modern retrieval uses embeddings — vectors that capture meaning so semantically-similar text matches even without shared keywords.
• RAG is the right tool for knowledge that changes. Fine-tuning is the right tool for behaviour and style that should stick.
• Most RAG failures are retrieval failures. Tune retrieval before you tune anything else.

RAG is one of three ways to shape what an LLM does for you. The others are prompting (ask better) and fine-tuning (change the weights). Knowing which to reach for is the last piece. Post 8 — and the close of this series.