Generator
robots.txt Generator
Most robots.txt files predate answer engines, so a wildcard written years ago now blocks them by accident. This builds an explicit file where every AI crawler is named, and training is a separate decision from answering.
- Every crawler namedNo accidental wildcards
- Training vs answeringTwo separate choices
- Audit it afterPairs with the auditor
Convert it
Paste it here
Your file
robots.txt is a request that well-behaved crawlers honour. It is a statement of policy, not an access control. Anything that must not be public needs authentication.
How to use it
Name every crawler explicitly
A wildcard rule written for scrapers now decides your AI policy silently. Naming each crawler means the file says what you actually chose, and the next person to read it can tell.
Blocking Google-Extended does not affect search
It governs Gemini training only. Teams block it believing they are protecting rankings and lose AI visibility for nothing. Googlebot is the token that matters for search.
Answering and training are different questions
You can reasonably want to be cited in answers while refusing to be training data. They use different tokens, so you can have one without the other.
This is a request, not a lock
Well-behaved crawlers honour robots.txt. Anything that genuinely must not be public needs authentication, not a Disallow line.
Questions
Blocking the AI-specific tokens does not affect traditional search crawling. It does remove you from AI answers, which is increasingly its own traffic source.
Run the result through the AI Crawler Access Auditor. It reports what your file actually does, including anything caught by a wildcard.
If you want to be cited, allow the answer engines. Training is a policy call and reasonable organisations decide it both ways.
Want this run against your whole site?
Tell us the page your assistant lives on and the questions you care about. A person runs them and sends back the raw answers.