Choosing what to train a website assistant on

When people set up a website assistant, the first instinct is to point it at the whole domain and upload every document they have. It feels thorough. It reliably produces worse answers than a carefully chosen subset.

The reason is mechanical. Retrieval works by finding the passages most similar to the question. Add more content and you add more candidates that are superficially similar but wrong. A blog post from two years ago describing a feature you removed will compete with your current documentation, and similarity does not know which one is true.

The three questions for every source

Is it current?

Outdated content is worse than absent content, because absent content produces “I do not know” and outdated content produces a confident wrong answer. Old pricing pages, deprecated feature docs and superseded policies should be excluded, not included and hoped over.

Is it authoritative?

For any given question there should be one place that answers it. If three pages describe your refund policy slightly differently, pick one and exclude the others. Contradictory sources produce inconsistent answers and there is no way to reason your way out of it at query time.

Is it prose?

Navigation menus, footers, cookie banners and legal boilerplate repeat on every page and add noise to every chunk. Content that is mostly structure rather than sentences is a poor retrieval source.

A starting set that works

For most marketing sites, this is the right initial scope:

• Product and feature pages, current versions only.
• Pricing page.
• Documentation or help centre, if it exists.
• Policy pages that visitors ask about: privacy, terms, refund, security.
• A hand written FAQ covering things not documented elsewhere.
• About page, for company questions.

Notably absent: the blog. Blog archives are the single most common source of bad answers, because they are large, old, exploratory and often contain positions the company no longer holds. Add specific evergreen posts by hand if they answer real questions. Do not add the archive.

The hand written FAQ is the highest value source

Every product has facts that live in someone's head and appear on no page. Whether you support a particular integration, what happens at a plan limit, whether a feature works on mobile. These generate questions constantly and there is nothing to retrieve.

Writing thirty of these as explicit question and answer pairs takes an afternoon and usually improves answer quality more than any other single change. It also has a useful side effect: the act of writing them tells you which ones should have been on your website all along.

Test with the questions you already have

If you have any record of what visitors ask, use those as your test set rather than inventing questions. Inventing test questions produces questions phrased the way you think, which is not how visitors think.

Twenty real questions, run against the assistant, read by hand. Count how many are answered correctly, how many are answered wrongly, and how many correctly say they do not know. The middle number is the one that matters, because a wrong confident answer costs more trust than a refusal.

Retraining cadence

Content changes and the index needs to follow. The rule that avoids both extremes: retrain when a source page materially changes, not on a fixed schedule. A weekly retrain on a site that changes monthly is wasted work. A quarterly retrain on a site that ships weekly means three months of wrong answers about pricing.

The pages worth watching most closely are pricing, plan limits and anything with a date in it.

The coverage and precision tradeoff

Every source you add increases what the assistant can cover and decreases how precisely it retrieves. These move in opposite directions and no setting reconciles them. What you are choosing is which failure you would rather have.

A narrow index refuses more often. On a dashboard that looks worse. Commercially it is usually better, because a refusal costs you one visitor who has to look elsewhere and a confidently wrong answer costs you a dispute conducted in writing.

The asymmetry is the whole argument. Refusals are recoverable and visible. Wrong answers are neither, because the visitor who received one does not report it, they just believe it.

A source list you can defend

Start from the sitemap, because it is a list you already maintain of pages you consider canonical. Then apply one rule: if you would not paste this URL into a reply to a customer, it is not a source. That single question typically removes a third of an over broad index, and it removes the third most likely to produce a confident error.

Remove anything dated. Blog archives, changelogs describing superseded versions, old policy pages that were never taken down. Content that was true and no longer is causes more damage than content that is absent, because absence produces a refusal and staleness produces an assertion.

Add the files that are not on the site at all. Handbooks, specification sheets, internal price lists that customers are allowed to see. These are frequently the highest value sources and they are invisible to a crawler.

Then give every source an owner. Not a product feature, a working practice. A source nobody is responsible for is a stale passage waiting to happen, and the date it goes wrong is the date somebody edits a page without telling anyone.

Auditing an index you inherited

If the index already exists and was built by pointing a crawler at the domain, run a pruning pass before changing anything else. It is an afternoon of work and it will outperform any model change.

Export the URL list. For each one answer three questions in writing. Would I send a customer this link today. Is anything on this page contradicted by another indexed page. Who notices if this goes out of date.

The second question is the one people skip and it is the most damaging. Two indexed pages that disagree will both be retrieved, and the assistant will answer from one without telling you it had a choice. Contradictions have to be fixed at the source, not mediated at retrieval time.

Repeat quarterly. An index is a maintained artefact, not a setup step, and its failure mode is always the same: it was correct on the day it was built.

Sources that seem obvious and are not

Four categories that most teams index by default and probably should not.

The blog archive. A post is a snapshot of an opinion on a date, written to be read as commentary, and an assistant will quote it as current fact. Index the handful that are genuinely evergreen reference material and leave the rest.

Job listings and about pages. Almost never asked about, frequently retrieved for questions about the company, and they crowd out real answers.

Case studies. Written to persuade rather than to inform, full of specific claims about particular customers, and dangerous when quoted as a general capability.

The changelog. Useful to a customer, actively harmful as a source, because it describes every state the product has ever been in and gives the retriever no way to know which is current.

Terms in this articleRetrieval Augmented GenerationChunkingRefusalIndex