Why multi model routing exists

The simplest possible architecture picks one model and sends everything to it. It is easy to reason about and it is what most implementations start with. It is also, past a certain volume, an unnecessarily expensive way to answer “what are your opening hours”.

Routing means choosing per query. Creobot is designed for multi model routing so teams can route conversations by cost, quality and task type.

The three axes

Cost

The spread between a small model and a frontier model is large, often an order of magnitude or more per token. On a marketing site the majority of questions are factual lookups against retrieved content, which is a task small models handle well. Sending those to a frontier model is spending capability on a problem that does not need it.

Latency

Smaller models respond faster. In a chat interface this is felt directly. A visitor waiting eight seconds for an answer that took two seconds to retrieve has a worse experience than the same answer delivered in three, even if the slower answer is marginally better written.

Task fit

Different tasks have genuinely different difficulty. Extracting a fact from a supplied passage is easy. Comparing two plans against a stated need, or reasoning about whether a described situation falls under a policy, is not. The second category benefits from a stronger model in a way the first does not.

How routing decisions get made

Three approaches, in increasing order of complexity:

**Rule based.** Classify by pattern. Short factual questions go to the small model, comparison and reasoning questions to the large one. Crude, transparent, and captures most of the available saving. Start here.

**Confidence based.** Try the small model first. If retrieval confidence is low or the model's answer is hedged, escalate to the larger one. Costs an extra call on escalated queries but routes on actual difficulty rather than surface features.

**Learned.** Train a classifier on which queries needed the larger model. Better in principle, requires labelled data you probably do not have yet, and is not worth building before the rule based version has told you the shape of your traffic.

The failure mode to design for

Routing introduces a new class of problem: inconsistency. The same question asked twice can be answered by different models and produce noticeably different responses. Visitors notice this, and it reads as unreliability.

Two mitigations. Route deterministically on question features so the same question routes the same way. And constrain output format tightly enough that model differences show up as phrasing rather than as structure or substance.

Fallback is the other reason to build this

Routing infrastructure gives you provider failover almost for free. If your primary provider returns errors or times out, the routing layer can send the request elsewhere rather than failing the conversation.

This is worth having independently of the cost argument. Single provider dependency means your assistant's availability is capped by theirs, and provider incidents happen. Design the routing layer so a fallback is a configuration change rather than a code change.

What not to claim

Routing does not make answers better in general. It makes the cost and latency profile better while holding quality roughly constant, and it adds resilience. Those are real benefits and they are not the same as improved accuracy. Accuracy comes from retrieval.

The reliability argument

Every model provider has incidents. This is not a criticism, it is what running infrastructure at that scale looks like, and it means an assistant with one provider has an availability ceiling set by somebody else's status page.

Without a fallback, a provider incident is a visible failure on your marketing site. The widget spins, or shows an error, and it does so during business hours because that is when load is highest for everyone.

With a fallback, the same incident is a slightly different tone of answer. Visitors do not notice. That difference is most of the argument for routing, and it has nothing to do with which model is better.

The economics

Most website questions are comprehension rather than reasoning. Reading three retrieved passages and phrasing an answer from them is not a hard task, and a mid tier model does it about as well as an expensive one.

Cost per conversation is the number that scales with success. An assistant that works gets used more, so a routing decision that looks like a rounding error at a hundred conversations a month is a real number at ten thousand.

The counterargument is consistency, and it is weaker than it sounds. Substance comes from the retrieved passages rather than from the model's own knowledge, so routing changes phrasing rather than facts. That is a benefit of grounding that is easy to miss.

Where routing does not help

It does not fix retrieval. If the wrong passage was retrieved, every model in the chain answers from it, and the more capable one produces a more convincing error.

It does not fix a bad system prompt. Behaviour that a buyer thinks of as model quality, refusing appropriately, citing sources, staying on topic, is mostly prompt design, and it applies identically whichever model answers.

It does not fix latency caused by retrieval. If the slow part is the vector search and the reranking pass, moving the generation to a faster model saves a fraction of the total and the visitor still waits.

What a visitor actually notices

Almost nothing about routing is perceptible, which is the point, and it makes the feature hard to sell and easy to under invest in.

What a visitor notices is time to first token. An answer that starts within a second and finishes in six feels faster than one that appears complete after three, because waiting with no feedback is longer than waiting with some.

They also notice inconsistency in tone across a single conversation, which is the one real risk of routing. Switching models mid conversation can change register noticeably. Routing per conversation rather than per message avoids it at almost no cost.

What they never notice is which model answered, which is why the honest framing of routing is operational rather than a quality claim.

A reasonable default

One capable primary, one fallback from a different provider, routing per conversation rather than per message, and a timeout tied to your latency budget. That covers the reliability case and most of the cost case without becoming a system nobody can reason about.

Terms in this articleRetrieval Augmented GenerationLatencyFallback ModelToken