Worksheet
Assistant Comparison Sheet
Vendor demos are run on questions the vendor chose. This produces one sheet, your questions, applied identically to every assistant you are considering — including the questions your content answers badly, which is where they differ.
- Your questions, not theirsSame set for every vendor
- Scores what mattersGrounding and citations
- Includes the hard onesWhere demos never go
Work it out
Your numbers
Your system prompt
Ask every assistant the same questions in the same order, in a fresh session, and record the answer verbatim. Paraphrasing while you score is how a comparison ends up confirming what you already believed.
What the numbers mean
The hard question is the whole comparison
Every assistant answers a well-documented question well. What you are actually choosing between is the behaviour when your content is thin, and no demo will show you that unless you bring the question.
Open the citation and read it
A citation that is present but does not support the claim is worse than none, because it looks verified. Scoring citations without opening them measures nothing.
Record verbatim, score afterwards
Paraphrasing an answer while forming an opinion of it is how a comparison confirms the conclusion you started with.
Include your current assistant
If you already run one, it belongs in the sheet. Sometimes the honest finding is that the incumbent is fine and the content is the problem.
Questions
Because it means driving several vendors' live widgets from a web page, which needs a backend and their permission. The sheet is slower and produces a result you can defend in a meeting.
About twenty minutes per assistant for the eight questions. The citation checking is the slow part and it is also where the differences appear.
The questions are yours and the scoring is yours. Q2 and Q3 are the rows we would want to be judged on, and they are equally hard for us.
Want this run against your actual site?
Tell us the page your assistant lives on and the questions you care about. A person runs them and sends back the raw answers.