RAG: giving a model memory it doesn't have

Post 6 ended on a hard limit: an LLM only knows what's baked into its weights plus whatever is in its context window right now. RAG is the most common way people get the right things into that window.

This is post 7 of 8 in the Foundations series.

What RAG actually means

RAG stands for Retrieval-Augmented Generation, which is a mouthful for three plain steps.

1. Retrieve. Take the user's question and search a corpus (your docs, your codebase, your wiki) for the chunks of text most relevant to it.
2. Augment. Drop those chunks into the LLM's prompt, right alongside the question.
3. Generate. Let the LLM answer, now that the fetched text is sitting in its context window.

That's the whole idea. No special model required. RAG runs on the same LLM you'd use without it. The only thing that changes is what's in the window when the model answers.

The word "retrieval" makes it sound like wizardry. It isn't. It's a search query whose results get pasted into the prompt ahead of the question. Every RAG system on earth, from a 100-line script to a sprawling enterprise stack, is a variation on that one move.

How retrieval actually works

The naive version of step 1 is keyword search. It's fine for some cases. The standard 2026 version is vector search built on embeddings.

An embedding is a list of numbers that captures the meaning of a piece of text. Two passages that mean similar things land close together in that number space, even when they share no actual words. "How do I cancel my subscription?" and "Stop my membership" embed right next to each other. A keyword search would sail straight past the second one.

The pipeline has two halves.

• Once, ahead of time: split your corpus into chunks (paragraphs, sections), run each chunk through an embedding model, and store the resulting vectors in a vector database (Pinecone, Weaviate, pgvector, or just a flat file if the corpus is small).
• At query time: embed the user's question, find the K nearest chunks in the vector store, and hand those back.

In practice, good systems run hybrid retrieval: vector search and keyword search together, with a re-ranker sitting on top. Pure vector search has its own bad habits, like cheerfully returning chunks that are related to the question without actually answering it. Most of the real engineering in production RAG is getting retrieval to be relevant and precise at the same time.

Why RAG isn't fine-tuning

This is the distinction that trips people up, so it's worth being blunt about it.

Fine-tuning changes the model's weights. You take an existing LLM and keep training it on your data until the new patterns are baked into the parameters. It's expensive, slow, and not easily undone short of rolling back to the old weights.

RAG doesn't lay a finger on the model. The weights stay frozen. Every query, the model is simply handed fresh text in its context, and it keeps no memory of the last query. Update your docs and RAG picks up the new version on the very next question. Fine-tuning would need a whole re-training run to do the same.

So the rule of thumb is clean. For factual knowledge that changes, RAG is almost always the right tool, because knowledge moves, re-training is costly, and updating an index is cheap. For changing the model's style, behaviour, or domain-specific reasoning, fine-tuning sometimes wins. Stable knowledge, reasoning patterns peculiar to your data, fixed output formats: those bake in well.

When RAG falls over

RAG is no silver bullet, and it fails in three recognisable ways.

Retrieval misses. If your search hands back the wrong chunks, the LLM has nothing real to ground in. It will either say "I don't know," if you're lucky and prompted it well, or hallucinate, if you weren't. Nearly every "my RAG is bad" complaint I've seen is a retrieval problem wearing an LLM costume.

Chunks are the wrong size. Tiny chunks lose their surrounding context. Huge chunks elbow other relevant material out of the window. There's no universal right answer here. Somewhere between 200 and 1,000 tokens is typical, tuned per corpus.

The model ignores the context. This is the lost-in-the-middle effect from post 6. Stuff 50 chunks into the prompt with the crucial one at position 25, and the model may walk right past it. Re-rankers and smarter ordering help.

A RAG system is only as strong as its weakest link, and the weak link is almost always retrieval. Fix that before you touch anything else.

RAG is one of three ways to shape what an LLM does for you. The other two are prompting (ask better) and fine-tuning (change the weights). Knowing which one to reach for, and when, is the last piece of the puzzle. That's post 8, and the close of this series.