Keeping an assistant in sync with a changing site

When you train an assistant on your website, you create a snapshot of that content at a point in time. The website keeps changing. The snapshot does not, until you retrain. Everything about staleness follows from that one fact.

The cost of staleness is not uniform

This is the part most retraining schedules ignore. A stale answer about your company story costs almost nothing. A stale answer about your pricing costs a customer and possibly a complaint.

So a single global retrain interval is the wrong shape. What you want is a tiering of sources by how expensive it is to be wrong:

• High cost: pricing, plan limits, availability, legal and policy pages. Retrain on change, same day.
• Medium cost: feature descriptions, integrations, documentation. Retrain weekly or on significant change.
• Low cost: about page, blog posts, general marketing copy. Retrain monthly or when convenient.

Change detection beats scheduling

A fixed schedule is either too frequent, wasting compute on unchanged content, or too infrequent, leaving known-wrong answers live. Detecting change is more efficient than guessing at an interval.

The cheap version, which works for most sites, is comparing a hash of the extracted text per URL. If the hash changed, that URL needs reindexing. If it did not, skip it. Content hashing rather than HTTP headers, because last-modified headers on many CMS platforms are unreliable and reflect template changes rather than content changes.

for url in sources:
 text = extract_text(fetch(url))
 h = sha256(normalise(text))
 if h != stored_hash[url]:
 reindex(url, text)
 stored_hash[url] = h

Normalising before hashing matters. Strip navigation, footer, and anything that changes on every page load such as a year in a copyright line, or every page will look changed on every check.

The deletion problem

Adding new content is the easy direction. Removing it is where systems commonly fail. If you delete a page describing a discontinued feature and only run an additive reindex, the old chunks stay in the index and the assistant keeps describing a feature that no longer exists.

Any sync process needs an explicit removal step: fetch the current source list, compare against what is indexed, and delete chunks whose source URL is no longer present or no longer returns a success status. Treat a 404 as a deletion signal, and treat a redirect as a move rather than a deletion.

What to do during a reindex

Reindexing is not instantaneous. During it, the index is in a mixed state. For a marketing site the honest answer is that this rarely matters, because a few minutes of mixed content is not a business risk.

For a pricing change it does matter. If you are changing prices, the sequence should be: publish the new page, retrain that specific source, verify with a test question, then announce. Not the reverse.

A verification habit

Keep a short list of test questions whose correct answers you know, covering your highest cost sources. After any retrain, run them. Five questions, thirty seconds, and it catches the case where a reindex silently failed and left you serving last quarter's prices.

Staleness is the failure mode nobody plans for

Teams plan for the assistant not knowing something. They rarely plan for it knowing something that used to be true, which is the more damaging case.

A refusal is visible and recoverable. A confidently outdated answer is neither. Retrieval worked, the model behaved correctly, and the output is a false statement delivered with total assurance because from the system's position nothing went wrong.

The gap between publishing a change and the index reflecting it is therefore a risk window, and its length is a decision you are making whether or not you think about it.

Scheduled versus triggered reindexing

Scheduled reindexing is simple, predictable and always slightly wrong. Between runs the index is a snapshot of the past, and the interval is how wrong you are willing to be.

Triggered reindexing fires on publish. It is correct, and it requires your publishing system to emit an event that something can listen to. A CMS usually can. A hand edited static site usually cannot, which is why the deployment method quietly determines the freshness guarantee.

The practical compromise is a schedule for everything plus a manual trigger for the pages where staleness is expensive. Pricing, policy, opening hours. Three pages, reindexed on change, removes most of the real risk.

Deletions are worse than edits

An edited page produces an outdated answer until the next index. A deleted page produces an answer citing a URL that now returns a 404, which is worse, because the visitor clicks the citation to verify and finds nothing.

That sequence destroys trust faster than a wrong answer alone, since it looks like the assistant invented a source.

Deletions should therefore trigger a reindex immediately rather than waiting for a schedule, and any citation to a URL that no longer resolves should suppress the answer rather than shipping it.

Detecting drift without waiting for a complaint

Keep a small set of questions whose correct answers you know and whose answers change when the site changes. Prices, limits, hours, policy terms.

Run them after every reindex and compare against the expected answer. This is a handful of questions and a few minutes, and it catches the case where a reindex silently failed, which otherwise surfaces weeks later.

Also watch citations pointing at pages you thought were removed. That is the clearest signal that the index and the site have diverged, and it is visible in any transcript log without additional instrumentation.

Who owns freshness

Every staleness incident traces back to nobody owning a page. The technical mechanism is rarely the cause; the cause is that somebody edited something and no process noticed.

Assign an owner per source, and make the list short enough that this is realistic. Thirty sources with owners beat three hundred without.

Then put reindexing where the publishing happens rather than where the assistant lives. A reindex triggered from your CMS on publish is part of someone's existing workflow. A reindex that requires logging into a separate tool is a step that gets skipped in the week it matters.

The pages worth this attention are few: pricing, policy, hours, anything contractual. Everything else can run on a schedule and the risk is acceptable.

The three pages worth a manual trigger

Pricing, policy and opening hours. Reindex those on change rather than on a schedule, and the overwhelming majority of expensive staleness disappears for almost no operational cost.

Terms in this articleIndex