Beyond vector search: hybrid, rerank, and graph RAG

Ask a naive RAG setup a plain factual question and it can come back with five confident, well-formatted, completely wrong chunks. Vector search did exactly what it was built to do, find text that sounds like the question. Sounding right and being right turn out to be two different jobs, and nothing in the system notices the difference on its own.

The previous post covered where naive top-k vector search breaks. This one covers what you actually reach for once it does. Three upgrades, in order of how often you'll need them.

Hybrid search: bring keyword matching back

Vector search finds things that mean something similar to your query. It's noticeably worse at exact matches, a product code, an error string, a person's name, because those don't have rich semantic meaning for an embedding model to latch onto. BM25, a keyword-ranking algorithm that predates embeddings by decades, is still better than vector search at exactly that.

Hybrid search runs both in parallel, a keyword search and a vector search, then combines the two ranked lists into one. You get vector search's sense of "similar meaning" and BM25's precision on exact terms, in the same result set. This is usually the first upgrade worth making, because it's cheap, well-understood, and fixes the most common naive-RAG complaint: "it can't find the thing I searched for by name."

Reranking: a second, smarter pass

Top-k retrieval pulls back a rough set of candidates fast, because comparing a query against millions of chunks has to be cheap. Reranking adds a second, slower pass: take that rough set, maybe 50 candidates, and re-score each one against the query using a model built specifically to judge relevance, not just similarity.

The reason this works is that the fast first pass and the careful second pass are optimizing for different things. The first pass has to scan everything, so it uses cheap approximate math. The second pass only has to compare 50 items, so it can afford to actually read each one carefully. You get speed from the first pass and precision from the second, instead of picking one.

Graph RAG: when the answer depends on relationships

Sometimes the fact you need isn't sitting in one chunk of text. It depends on how two things relate to each other, who reports to whom, which component depends on which, what happened before what. Flat chunk retrieval, however good the ranking, struggles here because it treats every chunk as an island.

Graph RAG retrieves from a knowledge graph instead, entities and the relationships between them, so a question about a relationship can be answered by walking the graph instead of hoping the right two chunks both got retrieved and the model correctly stitched them together. It costs more to build, since someone has to construct or extract that graph in the first place. It earns that cost specifically on relationship-heavy data, not on "what does section 4.2 say."

Agentic RAG: let the model decide when to search

Every pattern so far runs retrieval once, upfront, before the model ever sees the question. Agentic RAG flips that: the model gets a search tool and decides for itself when to use it, what to search for, and whether the results were good enough to answer with or worth a second search.

This matters for multi-part questions, where the first search reveals that a second, different search is actually needed, something a single fixed retrieval step can't adapt to mid-answer. It costs more per query, since the model might search two or three times instead of once, and it depends on the model being decent at tool calling, which is a real dependency, not a given.

Picking one

Start with hybrid search, it's the cheapest fix for the most common complaint. Add reranking when "the right chunk was retrieved, just not ranked first" is the failure you're seeing. Reach for graph RAG only when the questions are genuinely relationship-shaped. Agentic RAG is the right call when questions vary enough that a single fixed retrieval step can't cover all of them, and it's overkill if your questions are simple and consistent.

Retrieval solved doesn't mean the system is done, though. So far the model has only ever answered. The next four posts are about what happens once it starts acting.