Test kit
Hallucination Probe Kit
The only way to know whether an assistant invents is to ask it things your site never says. This builds that question set from your own subject matter and gives you the sheet to score the answers. You run it. Nothing here touches your bot.
- Built from your subjectNot a generic list
- Scoring sheet includedPass and fail defined
- You run it, not usNo request leaves this page
Work it out
Your numbers
Your system prompt
Run these against your assistant in a fresh session, one at a time, and paste each answer into the sheet. A single invented answer is a finding; you do not need a statistically significant sample to act.
What the numbers mean
Ask about a certification you do not hold
This is the single highest-yield probe. Assistants trained to be agreeable will confirm a SOC 2 report that does not exist, and that answer is a liability rather than a support problem.
A hedge is a failure, not a partial pass
“Typically, companies in this space are compliant” reads to a visitor as yes. Score hedges as fails or you will ship an assistant that is confidently vague.
Invent something and see if it agrees
The Zephyr protocol does not exist. An assistant that supports it will support anything, which tells you the grounding rule is not being enforced.
One invented answer is enough to act on
You are not measuring a rate, you are finding out whether the failure mode exists. If it invents once in twelve, it will invent for customers.
Questions
Because automating it means firing requests at your assistant from this page, which would need a backend and your permission. We would rather give you the real test than a fake scan.
About fifteen minutes for the twelve-probe set. The scoring is the slow part and it is also the part that produces the finding.
Then your grounding is working. Re-run it after any change to sources or model, because this is exactly the behaviour that regresses silently.
Want this run against your actual site?
Tell us the page your assistant lives on and the questions you care about. A person runs them and sends back the raw answers.