Auditor
AI Crawler Access Auditor
Most teams do not know whether their robots.txt lets answer engines read the site. Paste it in and this reports each crawler by name, including the ones a wildcard rule catches by accident.
- Fourteen crawlers checkedBy exact user-agent token
- Catches wildcard trapsThe accidental blocks
- Paste, never fetchYour file stays in the browser
Check it
Paste it here
Result
- Explicitly blocked —
- Blocked by a wildcard only —
- Training crawlers allowed —
- Answer engines allowed —
- Verdict —
- Notable —
A crawler blocked by a wildcard is usually an accident: the rule was written for scrapers years before answer engines existed, and nobody revisited it.
What the result means
Google-Extended does not affect Google Search
Blocking it removes you from Gemini training only. Teams block it believing they are protecting search rankings, and lose AI visibility for nothing. Search crawling is governed by Googlebot, which is a separate token.
A wildcard block is almost always an accident
“User-agent: * / Disallow: /” written years ago to stop scrapers now blocks every answer engine as well. If a crawler is blocked only by the wildcard, nobody decided that — it happened.
Training and answering are different decisions
You may reasonably refuse to feed a training corpus while still wanting to be cited in answers. They use different tokens, so you can have one without the other.
Being blocked is a position, not a bug
Blocking everything is legitimate if that is your policy. The failure is not knowing which you have chosen, which is what this tool is for.
Questions
Only from the ones that honour robots.txt. It is a request, not a wall. Treat it as a statement of policy that well-behaved crawlers respect.
No. It is parsed in your browser. Nothing is transmitted, and you can read this page's source to confirm it.
If you want to be cited in AI answers, allow the answer engines. Training crawlers are a separate policy call and reasonable people block them.
Want this run against your whole site?
Tell us the page your assistant lives on and the questions you care about. A person runs them and sends back the raw answers.