Create a robots.txt for AI crawlers
Tick who may read. The finished file is below, ready to copy, with one line of context per crawler.
Who may read?
1 crawlers will be blocked. Blocking means no longer appearing in those systems' answers.
robots.txt
# robots.txt — access for AI crawlers # Generated with geo-tool.com. Review before uploading. # ByteDance User-agent: Bytespider Disallow: / # Everyone else, including the AI crawlers not blocked above User-agent: * Allow: /
The file belongs in the root of your domain: your-domain.com/robots.txt. Do not overwrite existing rules for Googlebot or Bingbot — add the blocks above to them.
How this works
Order and specificity
A crawler follows the section carrying its own name. User-agent: * only applies when no dedicated section exists. That is why the generator writes a separate block per blocked bot and leaves a general Allow rule below — every other crawler stays untouched.
Add, do not overwrite
If you already have a robots.txt with rules for Googlebot, Bingbot or your own directory blocks, do not replace it with this file. Insert the generated blocks above and keep your existing rules.
Training and answering are two things
GPTBot reads for model training, OAI-SearchBot builds the index behind ChatGPT search. To stay out of training while still being cited in answers, block GPTBot and allow OAI-SearchBot. At Google, Googlebot and Google-Extended split the same two purposes.
Frequently asked questions
- What is the sensible default?
- Allow everyone. If you want to appear in AI answers there is no reason to block — except Bytespider, which is reported not to honour robots.txt reliably.
- Where does the file belong?
- In the root of the domain: your-domain.com/robots.txt. Subdirectories do not work.
- How long until a change takes effect?
- Crawlers usually re-read robots.txt daily to weekly. A new block takes days, not seconds.
- Do I also need an llms.txt?
- No, it is optional and not a Google standard. robots.txt governs access, llms.txt gives orientation. Both together beat either alone, but access comes first.
Check the whole page
This tool checks one aspect. The free GEO Check scores a whole URL across 25 checks in six categories. For pages behind a login, on staging or on localhost, the browser extension computes the identical score inside your own tab.