AI crawler checker
See in a few seconds which of the 12 most important AI bots your robots.txt allows, partly allows or blocks. With an explanation for each bot.
No account needed. We don't store what you enter or the result.
What this tool checks
- We fetch /robots.txt and read the rules according to the official standard (RFC 9309): the longest matching rule wins.
- For each bot we check whether it has its own group or falls under the general rule (User-agent: *).
- Search bots (such as OAI-SearchBot and PerplexityBot) fetch pages for AI answers. Training bots (such as GPTBot) collect text for model training; blocking them is a legitimate choice and separate from search.
- If your firewall or CDN blocks AI bots, this tool can't see that. The full scan does test it, with a real request as OAI-SearchBot, ChatGPT's search bot.
Want to know more?
Other free tools
This is one part. The full scan looks at more than 70.
The free Flowproof scan also checks your technical GEO, content, SEO and conversion, with concrete advice for every finding. A free account is all you need.