Rate limits, abuse and cost control

A website assistant is a public interface that triggers paid API calls. That combination means the cost is controlled by whoever visits, which is everyone, including automated traffic that has no interest in your product.

This is not a hypothetical risk. It is the predictable consequence of the architecture, and it should be handled at design time.

Layered limits

One limit is not enough because the abuse patterns differ. Four layers, each catching something the others miss:

• Per session. Caps a single conversation. Catches accidental loops and one determined person.
• Per IP per window. Catches scripted abuse from a single origin.
• Global per hour. The backstop. Catches distributed abuse that slips the first two.
• Per account monthly. The billing boundary, which is what the credit model in most products expresses.

The global limit is the one most often omitted and the only one that guarantees a bounded bill. Without it, a distributed source defeats the per IP limit and the cost is unbounded until someone notices.

What happens at the limit

This is a product decision that gets made by default. The options: stop and say so, degrade to a cheaper model, queue, or continue and bill overage.

For a free tier, stopping with a clear message is the honest behaviour. Silent degradation makes the product look bad without explaining why. Whatever you choose, say it on the pricing page, because “what happens when I hit the limit” is one of the most reliably asked pricing questions.

Input length is a cost lever

Cost scales with tokens, and a visitor can paste a very large amount of text into a chat input. Cap input length client side and validate server side, because client side alone is not a control.

Also cap the number of retrieved chunks passed as context. More context is not monotonically better for answer quality and it is monotonically worse for cost and latency. There is a point past which additional chunks add noise, and it is usually lower than people set it.

Caching the repeated questions

On a marketing site, question distribution has a heavy head. The same handful of questions arrive constantly. Caching answers to normalised repeat questions cuts both cost and latency meaningfully.

Two cautions. Normalise carefully, because two questions that look similar can differ in a way that changes the answer. And expire the cache when sources change, or you will serve last month's pricing from cache after publishing an update.

Alerting on spend

Set an alert at a fraction of your acceptable monthly spend, not at the limit. An alert that fires when you have already exceeded the budget is a report, not a control.

Alert on rate of change as well as absolute value. A tenfold increase in hourly requests is worth knowing about immediately, even when the monthly total is still comfortable, because by the time the monthly total is uncomfortable the incident has been running for days.

The three ways cost runs away

A scripted client hitting the endpoint in a loop. Rare but expensive, because every request is metered and the bill arrives before the monitoring does.

A single visitor treating the widget as a free model endpoint, asking it to write essays. More common than the first and harder to detect, because each request looks legitimate.

Retrieval returning too much. The quietest of the three: nothing is broken, no attacker is involved, every prompt is simply twice the size it needs to be, and the cost is double forever.

Limits that protect without punishing

Per session limits are better than per IP, because shared networks put many legitimate visitors behind one address.

A generous ceiling with a hard stop beats a tight one with an appeal process. Most real conversations are under ten messages, so a limit at thirty catches abuse and touches almost nobody.

Rate limit by burst as well as by total. Twenty messages over ten minutes is a conversation. Twenty in ten seconds is not.

And degrade rather than block. Slowing responses for a session that has exceeded a soft threshold is less likely to punish a legitimate power user than an outright refusal.

Controlling the cost nobody is attacking

The largest saving available to most deployments has nothing to do with abuse. It is retrieving fewer passages.

Ten retrieved passages instead of three roughly triples prompt tokens for an answer that is usually worse, because irrelevant context dilutes the relevant part. Fixing this improves cost and quality at the same time, which is unusual.

Routing simple questions to a cheaper model is the second lever, and it is invisible to visitors because substance comes from the passages rather than the model.

Caching identical questions is the third and it matters more than expected. On a marketing site the same five questions dominate, and answering them from cache is both instant and free.

Alerting on cost before the invoice

Metered spend is the one failure that arrives silently and lands as a number at the end of the month, by which point the cause is a fortnight old.

Alert on rate of change rather than on absolute total. A daily spend that doubles is worth knowing about at any level. A total threshold only fires once you have already spent it.

Track cost per conversation as the primary figure. Total cost rising because usage rose is success. Cost per conversation rising means something changed in retrieval, routing or prompt size, and that is the one worth investigating.

Keep a hard ceiling as a backstop, set well above normal, that stops requests rather than warning about them. Nobody wants to use it, and its absence is what turns a scripted attack into a large invoice.

Cache the questions everyone asks

On most marketing sites a small number of questions dominate. Answering those from a cache is instant and free, and it removes the largest share of metered spend before any limit is involved.

Key the cache on the normalised question plus the index version, so a reindex invalidates it and a stale answer cannot survive a content change.

Designing the limit message

Every rate limit eventually catches somebody legitimate, and what they see at that moment decides whether it reads as protection or as the product breaking.

Say that the conversation has reached its limit, not that something went wrong. The first is a boundary, the second sounds like a defect.

Give a route onward. A contact page, an email address, or a handoff. A visitor who hit a limit was engaged enough to send thirty messages, which makes them among the most interested people on the site that day.

Say when it resets if it resets. An unqualified limit reads as a permanent block.

And log every trip. A limit firing regularly for real visitors means it is set too low, and the only way to know is to look at the conversations that hit it rather than at the count.

Terms in this articleLatency