Handling questions the assistant should refuse

There is a temptation to treat every refusal as a failure. It shows up in the metrics as an unanswered question, and unanswered looks bad. That framing produces systems that answer things they should not, and a confidently wrong answer about a refund policy costs more than a refusal ever will.

The four categories that should always refuse

1. Account specific questions

“How many credits do I have left” cannot be answered by a system with no authenticated access to that account. Attempting it either fabricates or leaks. Route to login or to a human.

2. Commitments

Anything that would constitute a promise: custom pricing, contractual terms, delivery dates, whether you will build a feature. These need a person with authority, and a system that improvises them creates obligations the company did not agree to.

3. Legal, medical, financial advice

Even when your content touches these areas, applying them to someone's specific situation is a different act from describing them. State the general position, decline the application, point to a person.

4. Anything outside the indexed content

The broadest category and the one that matters most day to day. If no source covers it, say so. This is the boundary the whole design rests on.

Making refusal happen reliably

Instructing a model not to answer outside its context helps and is not sufficient on its own. Two structural mechanisms do more work:

A retrieval threshold. If the best matching chunk scores below a floor, do not pass context at all and return the refusal path directly. The model never gets the chance to over-reach because it never receives weak context to over-reach from.

Output constraints. Requiring the answer to cite which source it came from makes unsupported claims structurally awkward to produce and easy to detect afterwards.

Refusal wording matters

A refusal is still an interaction and it can be helpful or unhelpful. Three components make the difference:

• State plainly that the answer is not available, without apologising at length.
• Say what is available, if anything nearby is.
• Offer the next step, which is usually a person or a specific page.

"I do not have information about enterprise contract terms. I can tell you what is in the published plans, or connect you with someone who can discuss contracts." That is a useful refusal. It closes one door and opens two.

Tuning the threshold

Set the refusal threshold too high and the assistant is useless. Too low and it invents. There is no universal correct value, and it depends on your content density and embedding model.

The empirical approach: take fifty real questions, half of which you know are covered by your content and half of which are not. Sweep the threshold and find the point where covered questions are answered and uncovered ones are refused. Then set it slightly toward refusal, because on a public marketing site the cost of a wrong answer is higher than the cost of an unnecessary escalation.

Refusal is a feature, not a shortfall

An assistant that answers everything is either exceptionally well sourced or improvising, and on a marketing site the second is far more common than the first.

The reason to design refusal deliberately is that the cost of a wrong answer is asymmetric. A refusal costs one visitor a small amount of friction. A confident wrong answer about price, policy or capability is a written statement your company did not authorise.

Framing refusal as a quality signal rather than a limitation also changes how it is measured. Refusal rate becomes something to read rather than something to minimise.

The categories worth refusing outright

Anything requiring an account lookup. Order status, usage, whether a specific customer is on a specific plan. A content trained assistant has no access and guessing produces a cancelled order or a support escalation with an angry opening line.

Roadmap and unshipped features. This is where assistants invent most confidently, because marketing pages often describe aspiration in present tense and the assistant reads it as fact.

Anything contractual. Cancellation terms, liability, whether a clause applies. These need a person who can be held to the answer.

Competitor claims not present in your own published comparisons. An unsourced statement about another product is a liability rather than a sales asset.

Medical, legal or financial advice, on any site, regardless of what your content contains.

How to make refusal stick

Prompt instructions are strong defaults, not guarantees, and they degrade over a long conversation as the context fills with other material.

Anything with commercial consequence needs a second layer in code. A short list works better than a broad instruction: never state a price not present in a retrieved passage, never confirm a certification, never assert a contractual term, never claim availability of an unshipped feature.

Narrow rules enforced outside the model hold under pressure. Broad instructions to be careful do not.

Refusing without sounding useless

The wording carries most of the experience. A bare cannot help with that reads as broken. A refusal that names why, points somewhere real and offers a person reads as competent.

Say what it cannot do, in one clause. Offer the nearest thing it can do, if there is one. Offer the handoff. Three sentences at most.

Do not apologise repeatedly and do not blame the question. Visitors read excessive apology as evasion, and they read blame as exactly what it is.

Testing refusal before launch

Refusal is the half of behaviour that never gets tested, because a test set is naturally built from questions you want answered.

Write ten questions you want refused and keep them in the regression set permanently. A price for something not published. A roadmap question. A request for account specific information. A competitor claim. Something legal.

Run them after every content change, not just after configuration changes, because adding a source is the most common way a previously reliable refusal starts producing an answer. A marketing page written in aspirational present tense is enough to turn a refusal into a capability claim.

The pass condition is not that it declines politely. It is that it declines and offers the handoff, because a refusal with no route onward is where visitors give up on the product rather than on the question.

One line worth keeping

A refusal you designed is a boundary. A refusal that happens because retrieval failed is a content gap. They look identical to a visitor and should never look identical in your logs.

Terms in this articleRetrieval Augmented GenerationEmbeddingChunkingRefusal