Skip to content

AI crawler monitoring: can ChatGPT and Perplexity still read your site?

A robots.txt that allows everything doesn't mean AI bots actually get in. Flowproof checks the rules for 12 AI bots, makes a real request as OAI-SearchBot, ChatGPT's search bot, to find firewall and CDN blocks, and keeps an eye on it every week in Pro and Business.

Start for freeSee pricing
Feature
Included in: Free, Pro and Business
What it costs
Free to use with an account

Until 6 November, Flowproof is completely free: every category, including the accessibility check, for up to 3 websites and 25 scans a month.

What the check covers

What a finding looks like

An example from the report. The website is made up; the text is exactly what the scanner reports.

AI crawler access

  • ✕

    PerplexityBot (Perplexity)

    Blocked via the general rule (*). This bot indexes for Perplexity answers.

    Add to robots.txt:
User-agent: PerplexityBot
Allow: /
Without access, Perplexity can't cite you as a source.
  • !

    GPTBot (OpenAI)

    Blocked via its own rule. This bot collects content to train models.

    This is a training bot. Blocking it is a deliberate choice and does not count towards your score. Search bots have their own user agent and are checked separately.
  • ✕

    Server allows AI bots

    A request as OAI-SearchBot (ChatGPT's search bot) got status 403, while a regular browser does get access.

    Your firewall or CDN blocks AI bots, regardless of robots.txt. Turn off 'Block AI bots' in Cloudflare or add exceptions for the bots you want to allow.

Why monitor AI crawlers

Your website can only be a source for ChatGPT, Perplexity or Google AI Overviews if their bots can fetch your pages. That access can change without anyone touching the website. A security plugin gets an update, your hosting provider adjusts its bot filters, or your CDN rolls out new default settings. Your robots.txt looks exactly the same, but the bots get an error code.

That's why Flowproof checks on two levels. First, the scan reads your robots.txt according to the rules of the standard: the group naming the bot takes precedence over the general group, and within a group the longest matching rule wins. Then the scan makes a request with the user agent of OAI-SearchBot, ChatGPT's search bot, and compares the response with that of a regular browser. Background on all the bots and example files is in AI crawlers in robots.txt.

Search bots and training bots

Not every block is a mistake. Search and retrieval bots, such as OAI-SearchBot and PerplexityBot, decide whether you can appear in AI answers. Training bots, such as GPTBot and ClaudeBot, collect text to train models. Flowproof reports a blocked search bot as a problem. A blocked training bot is shown for information, with an explanation of the trade-off; it doesn't count in the score, because blocking it is a legitimate choice. OpenAI documents that OAI-SearchBot surfaces sites in ChatGPT search, that GPTBot is for training, and that you set the two independently. How AI systems then use the pages they fetch is explained in how AI search engines choose sources.

Weekly monitoring

In Pro and Business, Flowproof scans your websites automatically every week and sends an email report. You choose whether you always get it or only when something changes. That way you spot a new block within a week, not only once traffic from AI answers has dropped. The score history shows when the change began.

Part of the full GEO audit

The crawler check is one of the 9 areas of the GEO audit. Access is the first step; after that it's about understanding, with structured data and an llms.txt. To quickly test one domain without an account, use the free AI crawler checker. For the bigger picture, read the complete guide to GEO, or get started right away with a free account.

Updated on 18 October 2026

What it costs

All prices are per month, excluding VAT. Pay yearly and you get two months free.

Free

Find out whether AI can read your website.

€0forever

  • 1 website
  • 3 scans per month
  • Full advice on 4 core GEO areas
  • Overall score across all 9 areas
  • Comparison with your previous scan
Start for free

Pro

Most popular

For business owners who are serious about being found.

€190per year, excl. VAT

Average of €15.83 a month, billed annually

  • 5 websites
  • 100 scans per month
  • Full advice on all 9 areas
  • Score history and market benchmark
  • Weekly automatic check with email report
  • Print and export reports as PDF
  • Accessibility check against WCAG 2.2 (EAA and ADA)
Choose Pro

Business

For agencies and teams with multiple clients.

€790per year, excl. VAT

Average of €65.83 a month, billed annually

  • 25 websites
  • 1,000 scans per month
  • Everything in Pro
  • Client reports in your own branding
  • Early access to integrations for AI citations and brand mentions
  • Priority support
Choose Business

Frequently asked questions

Why isn't checking robots.txt enough?

Because the firewall or CDN decides before robots.txt does. If bot blocking is switched on in Cloudflare or at your hosting provider, an AI bot gets an error code, even when robots.txt allows it. Flowproof tests this with a real request as OAI-SearchBot.

Which AI bots should I allow?

If you want to appear in AI answers, at least allow the search bots: OAI-SearchBot, ChatGPT-User, Googlebot, Bingbot, PerplexityBot and Claude-SearchBot. Blocking training bots such as GPTBot and ClaudeBot is your own choice; Flowproof shows it for information and doesn't count it in the score. ChatGPT Search needs OAI-SearchBot, not GPTBot.

How often does Flowproof check my site?

You can scan whenever you like, within the number of scans in your plan. In Pro and Business, Flowproof also scans your websites automatically every week. You choose whether you always get an email or only when the score or findings change.

Can I test this without an account?

Yes, the free AI crawler checker shows what your robots.txt allows for each bot, no account needed. The firewall test, the noai check and the weekly monitoring are part of the scan with an account.

Further reading in the knowledge base

Background and explanation for this check, with sources.

Check it on your own website

Create a free account and scan your first website. You can upgrade any time.

Start for freeSee pricing
AI crawler monitoring: robots.txt and firewall tested