RAG done right: chunking and retrieval mechanics

People hear RAG and think: now the model can read my documents. That's not quite what happens. The model still only ever sees whatever text you hand it inside its context window, one time, for one answer. RAG is just a very deliberate way of deciding what that text should be.

The local RAG and embeddings post from the Local LLMs series got you a working setup: index a folder, embed it, search it, done. That version is enough to be useful and nowhere near enough to be reliable. This post is about the two decisions that actually decide whether retrieval works: how you cut the documents up, and what you do with the results once you've found them.

Chunking isn't a formatting step

Chunking means splitting a document into smaller pieces before you embed it, because you don't want to hand the model your entire 40-page PDF for a question about one paragraph. It sounds like a detail. It's actually one of the biggest levers on whether retrieval finds the right thing.

Three common approaches:

• Fixed-size chunking. Cut every N tokens, done. Simple, fast, and it will happily slice a sentence in half if the sentence happens to straddle a chunk boundary. The two halves get embedded separately, and neither half makes much sense on its own.
• Recursive chunking. Split on natural boundaries first, paragraphs, then sentences, only falling back to a hard cut if a paragraph is too long on its own. Respects the document's actual structure instead of a token counter.
• Semantic chunking. Group sentences by how similar their embeddings are to each other, so a chunk boundary lands where the topic actually shifts, not where an arbitrary counter hit zero. More expensive to compute, and it's the closest thing to "cut where a human would cut."

Chunk size itself is a trade-off, not a constant. Too large, and a chunk mixes several ideas together, so a search for one of them pulls back a bloated result full of irrelevant text riding along with the relevant part. Too small, and a chunk loses the surrounding context it needs to make sense on its own, a sentence like "it costs ₹2,000 more" is useless once separated from whatever "it" refers to. Somewhere in the 300 to 800 token range is the usual starting point, and the right number for your documents is something you actually have to test, not guess.

The embedding model is doing more work than it looks like

Every chunk gets converted into a vector by an embedding model, and every query gets converted the same way, so they can be compared. Two rules matter here. First, use the same embedding model for indexing and for querying. Switch models between the two and you're comparing vectors from two different, incompatible spaces, the search will return nonsense and nothing will look obviously broken. Second, general-purpose embedding models are fine for general text and noticeably worse on narrow domains, legal contracts, medical notes, your own codebase, where a model that's never seen that vocabulary struggles to place similar meanings close together in vector space.

Where top-k search quietly fails

The standard move is: embed the query, find the k chunks whose vectors are closest to it, hand those to the model. This works, and it also fails in a few specific, repeatable ways:

• Near-duplicate chunks crowd out real answers. If your documents repeat similar phrasing in five places, top-k can return five near-identical chunks and miss the one that actually answers the question, because "most similar" isn't the same as "most useful."
• The answer is split across chunks. If the fact you need requires combining information from two different sections, and only one of them gets retrieved, the model answers confidently with half the picture and no way to know it's missing the other half.
• Similarity isn't relevance. A chunk can be semantically close to the query's wording and still not answer the question. "Similar sounding" and "actually useful" are different properties, and vector search only measures the first one.

None of this means vector search is broken. It means naive top-k has a ceiling, and once you hit it, the fix isn't a bigger k, it's a different retrieval strategy. That's next.