Questions to ask any AI chat vendor before you sign

Evaluating this category is hard because the demos all look similar. Every product answers a question about your website convincingly in a controlled demo. The differences show up later, in the areas nobody demonstrates.

These are the questions worth asking. We have written them so they apply to us too, and where we do not yet have an answer, we say so.

Answer quality

• What happens when the assistant does not know? Ask them to demonstrate a refusal, not just a success.
• Can I see which source an answer came from?
• What happens if two of my pages contradict each other?
• How do I test it against my own real questions before committing?

That first one is the most revealing question in the list. A vendor who cannot readily show you a refusal either has not designed one or does not want to.

Content and control

• Can I exclude specific pages or sections?
• How does retraining work when my site changes, and is it automatic or manual?
• What happens to indexed content when I delete a page?
• Can I add answers by hand for things not on my site?

Data handling

• Which model providers do you use?
• Is provider training on my data disabled, and is that contractual or a setting?
• How long do you retain conversations, and how long do your providers?
• Where is data processed geographically?
• What is the deletion process and how long does it take end to end?
• Can I export everything?

Ask for these in writing. A confident verbal answer about retention is not the same as a documented one, and the difference matters if you are describing it on your own privacy page.

Escalation and operations

• Where does a handoff go, and what context travels with it?
• What happens outside working hours?
• What does the visitor see if your service is unavailable?

Commercial

• What is the unit, and what does it correspond to in real usage?
• What happens at the limit: stop, throttle, or overage billing?
• Can I move between plans in both directions?
• What is the notice period?

Compliance, and how to read the answer

If a vendor claims a certification, ask which auditor, which scope and when it was issued. A certification is a specific document about a specific scope, and “we are SOC 2 compliant” without those details is a marketing sentence rather than a fact you can rely on.

Equally, absence of a certification is not disqualifying for an early stage product. What matters is whether they are honest about where they are.

Our own answers, as of this writing

Applying the list to ourselves, in the interest of not asking others for a standard we do not meet:

• Creobot opens to teams in order, through the waitlist rather than open signup.
• Pricing shown on this site is a launch direction, not final production pricing.
• Model providers are not finalised. Creobot is designed for multi model routing so teams can route conversations by cost, quality and task type.
• Retention periods, subprocessors and data residency are not finalised and will be published before anyone is charged.
• We hold no security certification and are not claiming one.

If a vendor cannot produce an equivalent list, that is itself information.

Questions about grounding and refusal

What happens when the answer is not in my content. If the reply is that the model fills the gap from general knowledge, that is a product that will invent your pricing.

Can I see which passage an answer came from. If citations are not available, you have no way to debug a wrong answer and no way to tell retrieval failure from generation failure.

What is your refusal rate on a typical deployment. A vendor who has never measured it has not thought about the failure mode that matters.

Can I stop it answering specific categories, and is that enforced in code or only in a prompt. Prompt only enforcement is a default, not a guarantee.

Questions about content and freshness

How does chunking work. If the answer is a character count, expect answers cut in half.

How often is the index rebuilt, and can it be triggered on publish. The gap between the two is your staleness window, and it is the cause of the most damaging class of wrong answer.

What content cannot be reached. Every crawler has blind spots, and a vendor who claims none has not looked.

What happens to a page I delete. A citation pointing at a 404 destroys trust faster than a wrong answer, because it looks like the source was invented.

Questions about data, and the one that separates vendors

Who are your subprocessors. Ask for the list rather than an assurance. A vendor who cannot produce one either does not know their own chain or does not want to publish it.

How long are transcripts kept, and can I delete them. Retention is a product decision with legal consequences and the honest answer involves a tradeoff.

If you name a certification, ask which auditor, which scope and when it was issued. A certification is a specific document about a specific scope, and a claim without those details is a marketing sentence rather than a fact you can rely on. Equally, absence of a certification is not disqualifying for an early product, provided they say so plainly.

And the separating question: which of your claims are contractual and which are current practice. Practice changes without notice. It is uncomfortable to ask and more uncomfortable to answer, which is exactly why it is useful.

Two questions to ask yourself

What would make us remove this. Deciding the failure condition before purchase is what stops a disappointing tool surviving on inertia for a year.

Who will read the logs every month. If the answer is nobody, the assistant will degrade quietly regardless of which vendor you pick, because the content it depends on drifts and nothing surfaces that until a customer quotes a wrong answer back at you.

If a vendor will not answer

Treat a deflected question as an answer. Which auditor, which subprocessors and which claims are contractual are all questions with short factual replies.

The demo to insist on

A scripted demo tells you the product works on the vendor's content. That is not the question.

Ask them to index your site, live, and answer five questions you bring. Not questions you agreed in advance, and including at least two you expect to be refused.

Watch what happens on the refusals specifically. A vendor whose product invents an answer to a question your site does not cover has shown you the failure mode that matters, and no feature list compensates for it.

Watch the citations too. Ask which page each answer came from and check it. A confident answer citing a page that does not contain it is the single most informative thing you will see in any evaluation.

A vendor who declines this is telling you something. It takes them minutes if the product does what it claims.

Terms in this articleHuman HandoffRefusal