Latency budgets for a chat widget

When a visitor sends a message, several things happen in sequence, and the total wait is the sum. Knowing the breakdown tells you which part is worth attention, because they are not equally expensive to improve.

Where the time goes

• Embedding the query: usually tens of milliseconds. Not your problem.
• Vector search: single digit to low tens of milliseconds for a small index. Not your problem.
• Model time to first token: typically the largest single component, and the one that varies most by model.
• Token generation: proportional to answer length.
• Network and rendering: small, but it is where layout jank shows up.

The pattern is clear. Retrieval is fast and model time dominates. Optimising your vector index when the model accounts for most of the wall clock is misplaced effort.

Streaming changes the perceived number, not the real one

Streaming tokens as they generate does not reduce total time. It reduces time to first visible output, which is what the visitor actually experiences as responsiveness.

The difference is substantial in perception. A four second response that starts appearing after 700 milliseconds feels responsive. The same four seconds spent on a blank screen feels broken. If you build only one latency improvement, build streaming.

Answer length is a latency decision

Generation time scales with output length, so a verbose system prompt that produces long answers is also a slow system. On a website assistant, shorter answers are usually better anyway, so this is a case where the quality goal and the speed goal point the same direction.

Constraining answer length in the prompt is a legitimate latency optimisation and it costs nothing.

What to do while waiting

The interface during the wait matters more than most teams treat it. Three rules:

• Show something immediately on send, so the message is visibly received.
• Use a calm indeterminate indicator. Do not show a progress bar for something whose duration is unknown, because a bar that stalls reads worse than no bar.
• Never block the rest of the page. The visitor should be able to keep reading while the answer generates.

Timeouts and honest failure

Set a timeout and decide what happens at it. A request that hangs indefinitely is the worst outcome, because the visitor does not know whether to wait or leave.

At the timeout, say plainly that the response is taking longer than expected and offer the alternative, whether that is retrying or reaching a person. An honest failure preserves more trust than an indefinite spinner.

Measure at the edge, not at the server

Server side timing measures your infrastructure. It does not include network time to the visitor, which on mobile can dominate everything else. If you only measure server side you will conclude the system is fast while a meaningful share of your traffic experiences it as slow.

Measure from the client, and look at the slow tail rather than the average. The average visitor is not the one who leaves.

Where the time actually goes

A grounded answer has four costs and they are not evenly distributed. Embedding the question is fast, tens of milliseconds. Vector search is fast, similar. Reranking, if present, is slower. Generation dominates everything else.

This matters because the instinct when an assistant feels slow is to optimise retrieval, which is usually already the cheap part. The time is in the model producing tokens, and the only real levers there are a faster model or starting sooner.

The exception is a large index with no reranking, where retrieval returns a wide candidate set and the prompt becomes enormous. Then generation is slow because the input is long, and the fix is upstream after all.

Budget the number people feel

Time to first token is what a visitor experiences as speed. An answer that begins within a second and completes in six feels faster than one that appears complete after three, because waiting with feedback is shorter than waiting without.

A defensible budget on a marketing site is under one second to first token and under eight seconds to complete. Beyond that, abandonment rises sharply and the visitor has usually started doing something else.

Streaming is therefore not a nicety. It changes perceived speed far more than it changes real speed, which is why it is worth implementing even when total time is unchanged.

What to do when it is genuinely too slow

Retrieve less before you generate faster. Two or three well ranked passages beat ten, and they beat them on quality as well as speed, so this is the change with no downside.

Route simple questions to a faster model. Most website questions are comprehension rather than reasoning, and the difference in answer quality on those is not perceptible while the difference in latency is.

Move reranking behind a threshold. Run it only when the top candidates score closely, which is when the ordering is actually uncertain and reranking earns its cost.

And show something immediately. An acknowledgement while retrieval runs converts dead air into a wait, which people tolerate far better even when the elapsed time is identical.

Measuring it honestly

Server side timing understates what a visitor experiences, because it excludes the network, the widget's own startup and the time before the first character is painted.

Measure from the client, from the moment the visitor sends to the moment the first character appears. That is the number that correlates with abandonment.

Record the distribution rather than the average. An average of two seconds hides a tail where one conversation in twenty waits nine, and it is the tail that people remember and talk about.

Watch the ninety fifth percentile specifically after any change to retrieval or source count. Adding sources tends to move the tail long before it moves the average.

Perceived speed is cheaper than real speed

Streaming, an immediate acknowledgement, and a visible indication that retrieval is running cost almost nothing and change the experience more than shaving a second off generation.

Do those first. They are reversible, they do not affect answer quality, and they buy time to decide whether the real latency is worth engineering effort.

The budget in parts

Writing the budget down as a set of allowances rather than a single number makes it possible to tell which part broke when the total moves.

A workable split on a marketing site: fifty milliseconds to embed the question, a hundred for vector search, two hundred for reranking if present, and the remainder for generation up to the first token.

That leaves roughly six hundred milliseconds of generation latency inside a one second budget, which most models meet for a short prompt and few miss unless the prompt is large.

Prompt size is therefore the variable to watch, and it is set by how many passages you retrieve. Ten passages instead of three is the most common reason a budget that worked at launch stops working three months later, and it usually happens because someone added sources rather than because anything was tuned.

Measure each part separately at least once. Teams who only have the total spend their optimisation effort on whichever part they assume is slow, and the assumption is wrong about half the time.

Terms in this articleRetrieval Augmented GenerationEmbeddingLatencyVector DatabaseSystem PromptToken