Retrieval augmented generation, a practical explanation

Retrieval augmented generation is a plain idea wrapped in an intimidating name. Instead of expecting a language model to know your content, you look up the relevant passages at question time and hand them to the model along with the question.

The model's job becomes reading comprehension rather than recall. That is a much easier job, and it is why this approach works at all for content the model has never seen.

The pipeline, concretely

1. Split your content into chunks, typically a few hundred words each.
2. Convert each chunk into a vector, a list of numbers representing its meaning.
3. Store the vectors in an index.
4. At question time, convert the question into a vector the same way.
5. Find the chunks whose vectors are closest to the question vector.
6. Put those chunks and the question into a prompt.
7. The model answers using the supplied text.

Where quality actually comes from

Step five is where most answer quality is won or lost, and it is the step teams spend least time on.

If retrieval returns the right passage, a mid tier model will answer correctly. If retrieval returns the wrong passage, the best model available will answer confidently and wrongly, because from its perspective it was given the relevant context and asked to use it. It has no way to know that the retrieval step failed.

This is the practical consequence: when answers are bad, check what was retrieved before changing the model. Most of the time the model did a reasonable job with bad input.

Why similarity is not relevance

Vector search finds text that is semantically similar to the question. Similar is not the same as containing the answer.

Ask “how much does the Pro plan cost”. A chunk that discusses pricing philosophy at length will score highly on similarity because it is full of pricing vocabulary. The pricing table, which is mostly numbers and short labels, may score lower despite containing the actual answer.

This is a known and structural weakness. Sparse or keyword based retrieval handles the table well and handles paraphrased questions poorly. Combining both, sometimes called hybrid retrieval, covers more cases than either alone.

The refusal path is part of the design

If nothing relevant is found, the correct behaviour is to say so. This has to be built in deliberately, because the default behaviour of a model handed weakly relevant context is to use it anyway.

Two mechanisms help. A similarity threshold below which you do not pass context at all, and an explicit instruction that answering from outside the supplied context is not permitted. Neither is perfect. Together they turn most would-be fabrications into an honest refusal, which is the outcome you want on a website where a wrong answer about pricing or policy is worse than no answer.

What to measure

Two numbers, measured separately, because they have different fixes:

• Retrieval hit rate: for a set of questions whose answers you know exist, how often does the correct chunk appear in the retrieved set. This isolates step five.
• Answer correctness: given correct retrieval, how often is the final answer right. This isolates the model and the prompt.

Measuring only the second conflates the two and sends you shopping for a better model when the fix was chunking. Build a set of twenty real questions with known answers and check both. It is an afternoon of work and it will change where you spend the next month.

Where quality is actually won and lost

There are seven steps in a retrieval pipeline and they do not contribute equally. Chunking and source selection account for most of the variance in answer quality. The model, which is the part everyone argues about, accounts for very little once you are above the cheapest tier.

This is counterintuitive because the model is the part that produces the visible sentence. When an answer is wrong the model wrote the wrong sentence, so the model looks responsible. It usually was not. It was handed a passage and asked to answer from it, and it did exactly that.

The practical test takes ten seconds. Look at the citation on a wrong answer. If the cited page is wrong or out of date, retrieval worked and your content did not. If the cited page is right and the answer misreads it, that is the rarer case where the model is genuinely at fault.

Why chunking on character count breaks answers

The simplest chunking strategy splits text every N characters. It is easy to implement, produces evenly sized passages, and quietly destroys a proportion of your answers.

The failure is structural. A question and its answer usually sit within one section of a document. A character boundary falling in the middle of that section produces two passages, one holding the setup and one holding the resolution. Neither answers the question. Retrieval will return whichever is more similar to the question, which is usually the setup, so the assistant reads a passage that poses the question it was asked and answers from nothing.

Splitting on document structure instead, following headings and paragraph boundaries, produces uneven passage sizes and much better answers. Uneven is fine. Vector search does not require uniform lengths, and the tidiness of equal chunks is aesthetic rather than functional.

The retrieval failures nobody warns you about

Similarity is not relevance. Two passages can be semantically close to a question and only one can answer it. A page describing a feature and a page announcing that the feature was deprecated will both match a question about that feature, and the deprecation notice is often shorter and therefore less similar.

Recency is not encoded. An embedding carries meaning, not time. Nothing in the vector says one passage is current and another is two years old, so an outdated page competes on equal terms and sometimes wins.

Retrieving the page rather than the passage is the third. Some implementations return whole documents. On a long page this fills the context window with mostly irrelevant text, which dilutes the relevant part and makes the answer worse the longer the page is.

How to evaluate a pipeline before it goes live

Write down twenty questions you know the answers to, drawn from real support threads rather than invented. Include five you expect the assistant to refuse, because refusal behaviour is the half most teams never test.

Run them and record two things per question: whether the answer was correct, and which passage was cited. The second is the diagnostic. A correct answer from the wrong passage is luck and it will not hold.

Then change one thing at a time. Chunking strategy first, since it has the largest effect. Source list second. Model last, and usually not at all. Changing several at once means you learn nothing from the result, which is how teams end up believing the expensive model fixed something.

Terms in this articleRetrieval Augmented GenerationChunkingVector DatabaseRefusalIndex