Reliability when a model provider goes down

Every model provider has incidents. Rate limits tighten under load, latency spikes, endpoints return errors. If your assistant calls one provider with no alternative path, its availability is bounded by that provider's, and you have no lever.

The four failure shapes

They need different handling, and treating them identically is the usual mistake:

• Hard errors. A 500 or a connection failure. Fast to detect, and retryable.
• Rate limiting. A 429. Retrying immediately makes it worse. Needs backoff.
• Latency degradation. Requests succeed but slowly. The hardest to detect because nothing is technically failing.
• Quality degradation. Responses return normally and are worse. Effectively undetectable in real time.

Most retry logic handles the first and mishandles the second. Backoff with jitter is not optional on a 429, because synchronised retries from many clients are what turn a brief limit into a sustained one.

Fallback ordering

A sensible degradation ladder, in order:

1. Retry the same provider with backoff. Handles transient errors.
2. Route to a secondary provider. Handles sustained provider issues.
3. Route to a smaller or cheaper model on any available provider. Degraded but functional.
4. Return retrieved content without generation. The visitor sees the relevant passage rather than a synthesised answer.
5. Honest failure with a handoff offer.

Step four is underused and it is genuinely valuable. Retrieval usually still works when generation does not, because it depends on different infrastructure. Showing the relevant passage from your own documentation is a real answer, just an unpolished one.

Timeouts have to be set explicitly

Default HTTP timeouts are typically far longer than a chat interaction can tolerate. A visitor is not waiting sixty seconds. Set the timeout to what the experience can bear, which is usually well under ten, and treat exceeding it as a failure to be handled rather than a request to keep waiting for.

Circuit breaking

If a provider is failing, continuing to send it traffic wastes time on every request and delays every visitor. A circuit breaker that trips after a threshold of failures and routes elsewhere for a cooldown period converts a slow degraded experience into a fast degraded one, which is better.

The parameters matter less than having one at all. Trip after a handful of consecutive failures, cool down for a minute, then probe with a single request before restoring.

What to tell the visitor

During degradation, say something true and specific. "We are having trouble reaching our AI service. You can leave a message and we will follow up, or try again in a few minutes." That is better than a generic error and much better than a spinner that never resolves.

Do not claim the answer is being generated when no request is in flight. Visitors sometimes wait for several minutes on the strength of an indicator that is lying to them.

What to measure

Error rate and latency per provider, tracked separately, with the slow tail rather than the average. And fallback activation rate, because if you are silently falling back constantly, your primary configuration is wrong and nobody would otherwise notice.

What an outage looks like from a visitor's side

Rarely a clean error. More often a request that hangs, then times out, then leaves a widget showing a spinner that never resolves.

That is worse than an explicit failure, because the visitor cannot tell whether to wait. A clear message saying the assistant is unavailable and here is how to reach a person is a recoverable experience. A spinner is not.

Partial degradation is the harder case. A provider that is slow rather than down produces answers eventually, and eventually is past the point where the visitor left.

Designing for it before it happens

A fallback provider is the main defence, and the important detail is that it must receive the same retrieved passages and the same grounding instruction. Falling back to a model with different behaviour turns an availability incident into an accuracy incident.

Set a timeout that reflects the latency budget rather than the provider default. Waiting thirty seconds for a primary before failing over means the visitor has already gone.

Fail over per request, not per deployment. A provider having a bad minute should not require a configuration change.

And make the failure path visible in your own logs, or you will discover the fallback has been carrying all traffic for a week.

What to tell people during an incident

Say the assistant is unavailable, not that something went wrong. The first is information, the second is noise.

Offer the route that still works. A contact page, an email address, a phone number. The point of the assistant was to answer a question, and that need does not disappear because a provider is down.

Do not queue the question with a promise to answer later unless something actually will. An unfulfilled promise during an outage is remembered longer than the outage.

Testing the fallback before you need it

A fallback that has never been exercised is a hypothesis. Most fallback paths are written once and first executed during a real incident, which is the worst possible time to discover a configuration error.

Force it deliberately on a schedule. Point the primary at an invalid endpoint for five minutes in a staging environment and confirm the fallback answers, with the same passages and the same refusal behaviour.

Check the answer quality, not just that something came back. A fallback that responds without grounding is worse than an outage, because it produces confident invented answers at exactly the moment nobody is watching.

And confirm your monitoring notices. If the failover is silent, you will not know that the primary has been down for a week.

Status pages are not monitoring

A provider status page updates after the provider has acknowledged an incident, which is routinely later than your visitors noticed.

Monitor your own error and timeout rate instead. It moves first, it moves for degradation as well as for outages, and it catches problems that are yours rather than theirs.

Degradation is more common than failure

Complete outages are rare and obvious. Slow periods are frequent and quiet, and they cause more lost conversations in aggregate.

A provider under load does not usually error. It responds, eventually, at three or four times normal latency. Every request succeeds, no alarm fires, and your visitors leave before the first token arrives.

This is why a timeout tied to your latency budget matters more than an error handler. If your budget is one second to first token, a request that has not started responding in two seconds should move to the fallback regardless of whether the primary would eventually have answered.

It is also why the metric to watch is the ninety fifth percentile rather than the average. Degradation shows up in the tail long before it moves the mean, and by the time the mean moves the problem has been live for hours.

Log every failover with its reason. A week of failovers on timeout rather than error is a provider quietly getting slower, which is information you want before it becomes an outage.

Terms in this articleRetrieval Augmented GenerationHuman HandoffLatencyFallback Model