Where do I find my robots.txt?

At your-domain.com/robots.txt. If nothing is there, no file exists - and no crawler is blocked.

Should I block AI crawlers?

If you want to appear in AI answers, do not block them. Blocking makes sense when content must not be used for model training and visibility in AI answers is secondary. Having both is only partly possible: you can allow OAI-SearchBot while keeping GPTBot blocked.

Does blocking hurt my Google ranking?

No, as long as Googlebot stays allowed. According to Google, Google-Extended is a separate token that only affects Gemini training and grounding, not Google Search or ranking.

Is my robots.txt uploaded anywhere?

No. The check runs entirely in your browser; nothing is sent and nothing is stored.

Free tools

Does your robots.txt block GPTBot and friends?

Paste your robots.txt. The check tells you for every relevant AI crawler whether it may read the page. It runs in your browser - nothing is uploaded.

Contents of your robots.txt

Found at your-domain.com/robots.txt. The text stays in your browser.

Result

Paste your robots.txt above - the result appears immediately.

How this works

What counts as a block

Only a full block counts, meaning Disallow: / - either in the bot’s own section or via User-agent: * without a milder exception for it. Individual disallowed directories are not an exclusion: the crawler may still read everything else.

The Google-Extended trap

Google-Extended only controls whether content is used to train Gemini models and to ground Gemini Apps. Blocking that token does NOT remove you from Google Search or from AI Overviews: both run on Googlebot. To keep content out of AI Overviews you need snippet controls such as nosnippet or max-snippet, not Google-Extended.

robots.txt is a request, not a lock

The file is a voluntary agreement. Reputable operators honour it, others do not - Bytespider has been reported to ignore it repeatedly. To truly lock a crawler out, block it at server or CDN level as well.

What open access does not yet mean

A crawler that may read still finds nothing if the content only appears through JavaScript. No major AI crawler executes JavaScript. Access is the precondition, not the result.

Frequently asked questions

Where do I find my robots.txt?: At your-domain.com/robots.txt. If nothing is there, no file exists - and no crawler is blocked.
Should I block AI crawlers?: If you want to appear in AI answers, do not block them. Blocking makes sense when content must not be used for model training and visibility in AI answers is secondary. Having both is only partly possible: you can allow OAI-SearchBot while keeping GPTBot blocked.
Does blocking hurt my Google ranking?: No, as long as Googlebot stays allowed. According to Google, Google-Extended is a separate token that only affects Gemini training and grounding, not Google Search or ranking.
Is my robots.txt uploaded anywhere?: No. The check runs entirely in your browser; nothing is sent and nothing is stored.

Check the whole page

This tool checks one aspect. The free GEO Check scores a whole URL across 37 checks in six categories. For pages behind a login, on staging or on localhost, the browser extension computes the identical score inside your own tab.

Run the free GEO Check Browser extension