Robots.txt checker for Google and AI crawlers
The Rankmetr robots.txt checker reads your file, shows which rules apply to Googlebot and to AI crawlers, and tests whether those bots can actually reach your pages. The tool is in early access: leave your website and we will tell you when the free check opens.
Check website robots txt
The free robots txt checker is in early access. Enter your website and work email and we will let you know when you can run it.
What the robots.txt checker looks at
- The live file: fetches yoursite.com/robots.txt the way a crawler does, including rules added by your CMS or CDN that you never wrote.
- Groups and rules: parses each User-agent group with its Allow and Disallow rules, wildcards and the Sitemap line, and points out syntax errors like a robots.txt validator.
- Search crawlers: whether Googlebot can crawl each key URL on your domain, so the robots.txt checker covers Google as well.
- AI crawlers: which rules apply to GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended: allowed, blocked or partly blocked.
- Real access: fetches your pages with each bot's User-Agent to see if a firewall or bot protection blocks it even when robots.txt allows it.
How to read the result
A crawler follows only one group: the one that matches its name most specifically. If your file has a group for GPTBot, the bot ignores the rules under User-agent: *. This is the most common surprise in a robots.txt file: a wildcard block that you think covers everyone does not apply to a bot with its own group.
Allowed means the bot can crawl the site. Blocked means a Disallow: / rule closes the whole site. Partly blocked means some folders are closed, so check that no key URL is among them. If the file allows a bot but the real request fails, the block is in your hosting or CDN settings, not in robots.txt.
AI crawlers you should know
GPTBot
OpenAI crawler that collects content for model training. Blocking it does not remove you from ChatGPT search, which uses OAI-SearchBot.
ClaudeBot
Anthropic crawler for its models.
PerplexityBot
Crawler that Perplexity uses to find and cite pages in its answers.
Google-Extended
A control token, not a separate bot. It tells Google whether content may be used for Gemini. Google says it does not affect ranking in Search.
What to do with the results
- 01
Decide on AI access
If you want to appear in AI answers, do not block the search bots of AI assistants. Training bots are a separate choice.
- 02
Fix the file
Close only service folders. Never block CSS or JS files that pages need to render.
- 03
Check the CDN
If bots are blocked by bot protection, add exceptions in your CDN or hosting settings.
- 04
Re-check after changes
Run the check again after each site update; a site audit does it on a schedule.
llms.txt vs robots.txt
robots.txt sets access rules for crawlers. llms.txt is a proposed file that describes your key pages for language models; it does not grant or block access, and its effect is not proven. Get robots.txt right first, then create the file with the llms.txt generator if you want it. For the bigger picture of visibility in AI answers, see GEO.
FAQ
What does a robots.txt checker do?
It fetches the robots.txt file of a website, parses its rules and shows which bots can crawl which parts of the site. A good AI crawler checker also tests real access with each bot's User-Agent.
Is a robots.txt checker the same as a validator?
Close, but not the same. A validator checks syntax, like a file you paste in. A checker also tells you what the rules mean for specific crawlers and pages.
Why is an AI crawler allowed when my wildcard group blocks it?
Because the bot has its own group in the file. A crawler follows the most specific group that matches its name and ignores the User-agent: * rules.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects data for training. ChatGPT search uses OAI-SearchBot, which has its own rules.
Does Google-Extended affect my Google rankings?
Google says it does not. It controls the use of your content for Gemini, not crawling for Search.
Does robots.txt remove pages from Google?
No. It blocks crawling, not indexing. A blocked page can still appear in results if other sites link to it. Use a noindex tag to remove a page.
Get early access to the robots.txt checker
Leave your website and work email. We will tell you when the free check opens.