Measuring answer quality without a labelled dataset

Every guide to evaluating a retrieval system assumes a labelled dataset: questions paired with correct answers, ideally hundreds of them. Almost nobody has this when they need it, and building one is exactly the kind of work that gets deferred forever.

Two proxies are available immediately and cost nothing to collect.

Proxy one: unanswered rate

The share of questions where the assistant said it could not answer. This is a directly observable number and it moves for readable reasons.

It is a proxy for coverage rather than accuracy, and the relationship to quality is not monotonic. A very low unanswered rate is not good news; it usually means the refusal threshold is too permissive and the system is answering things it should not.

What you want is for the number to be stable and for changes to be explainable. A jump after a content change means something broke in indexing. A jump with no change on your side means traffic composition shifted.

Proxy two: human correction rate

Among conversations that reached a human, how often did the human have to correct something the assistant said.

This is the strongest available signal for accuracy because it is a real judgement by someone who knows the answer, generated as a byproduct of work that was happening anyway. It requires one small process change: the person handling the escalation marks whether a correction was needed.

One checkbox. That is the entire instrumentation cost, and it produces a labelled dataset over time as a side effect.

Building the golden set incrementally

Every corrected answer is a labelled example. Question, wrong answer, correct answer. After a few months of collecting these you have the dataset you could not justify building upfront, and it is drawn from real traffic rather than from someone's imagination of real traffic.

This is the practical path: do not build the eval set first, build the mechanism that generates it.

What thumbs up and down does not tell you

Inline feedback widgets have a low response rate and a strong selection bias. People rate when they are annoyed. A thumbs down rate is closer to a frustration measure than an accuracy measure.

It is not useless. Individual thumbs down conversations are worth reading because they point at specific failures. But the rate should not be presented as an accuracy metric, and comparing it across time is unsound because the population rating it changes.

Separating retrieval failure from generation failure

When you review a bad answer, the diagnostic question is which step failed. Look at what was retrieved.

• Correct chunk retrieved, wrong answer produced: a generation or prompt problem.
• Wrong chunk retrieved: a retrieval problem, usually chunking or source selection.
• Nothing relevant existed to retrieve: a content problem, and the right fix is writing the missing page.

These have completely different fixes and the third is the most common on a new deployment. Logging the retrieved chunk IDs alongside each answer costs almost nothing and makes this diagnosis take seconds instead of guesswork.

Why the obvious approaches fail

Thumbs up and thumbs down looks like the answer and is not. Response rates are low, and the people who respond are disproportionately those who had a strongly bad experience, so the signal is both sparse and skewed.

Manual labelling works and does not scale. Grading a hundred answers is a useful afternoon. Grading them weekly is a job nobody keeps doing after month two.

Using a model to grade another model's answers is circular in a specific way that matters: the grader has the same blind spot about retrieval. It will mark a fluent answer from the wrong passage as correct, because it is only reading the answer.

Four proxies that hold up

Refusal rate by topic. Not a failure metric. A rising refusal rate in one area is a measurement of a content gap, and it is more reliable than any accuracy score because refusal is unambiguous.

Follow up rate. How often a visitor asks again immediately after an answer. A high follow up rate on one topic almost always means the answer was technically correct and practically useless, which no accuracy measure catches.

Escalation rate split by trigger. Escalations from repeated failure indicate quality problems. Escalations from explicit request usually do not, and conflating them hides both.

Abandonment after answer. The visitor received an answer and left the site within a few seconds. Ambiguous on its own, informative in aggregate when it clusters on one topic.

The one measurement worth doing by hand

Citation correctness. Take thirty answers, look at the passage each one cited, and record whether that passage actually contained the answer.

This is the highest information per minute check available, because it separates the two failure modes that every other metric conflates. An answer can be wrong because retrieval returned the wrong passage or because the model misread the right one, and the fixes are completely different.

In practice most errors are the first kind, which is why this check so often redirects a team away from model tuning and toward their own content.

Building a regression set from real questions

Once you have read enough transcripts, pull twenty questions into a fixed set and rerun them after every content or configuration change.

Include five you expect to be refused. Refusal behaviour is the half nobody tests, and it is the half that protects you from writing commitments you did not authorise.

Record the cited passage alongside the answer each time. When something regresses, the citation diff tells you what changed far faster than rereading the answers.

Reporting it to people who did not build it

Proxies are defensible internally and hard to present externally, because none of them is an accuracy figure and somebody will ask for one.

The honest framing is to report direction rather than level. Refusal rate on this topic fell after we published that page. Follow up rate on pricing questions halved after we added the limits section. These are claims the data supports.

Resist producing a single quality score. Any number that combines these proxies into one figure hides which one moved, and it will be quoted back at you in six months as if it were measured.

If someone insists on accuracy, the answer is the manual citation check on thirty answers. It is a real number with a stated sample size and a known method, and it is worth more than a dashboard figure nobody can define.

Terms in this articleRetrieval Augmented GenerationChunkingRefusal