AI crawlers in robots.txt: should you block or allow GPTBot?

Whether you should block or allow GPTBot comes down to one question: do you want your content to be used to train AI models? GPTBot is a training bot. Your visibility in ChatGPT's search answers depends on a different bot, OAI-SearchBot. If you don't make that distinction, it's easy to block the wrong bot and disappear from AI answers without noticing.

This article is a practical how-to that is part of our complete guide to GEO. You'll get an overview of all the major AI crawlers, two ready-to-use example files and an explanation of how robots.txt applies its rules. After that, we cover a common trap: a robots.txt that allows everything while the firewall still blocks the bots.

#Search bots and training bots do different things

AI bots fall into two groups: search and fetch bots, which decide whether you appear in AI answers, and training bots, which collect content to train models. A search bot builds an index from which an AI search engine retrieves and cites sources. A fetch bot visits a page at the moment a user asks for it. A training bot collects text for future model versions, with no source attribution in return.

If you block a search bot, an AI search engine can no longer use your page as a source. If you block a training bot, the documentation of the major companies says nothing changes about your visibility in their search features. You can read more about how that source selection works in our article on how AI search engines choose sources.

#Overview of AI crawlers by company

The table below lists the AI bots of the largest companies, with their purpose and the effect of blocking them. All data comes from the official documentation, linked for each company.

The sources: OpenAI on its crawlers, Google on its crawlers and AI features in Search, Anthropic on its crawlers, Perplexity on its bots, Apple on Applebot, Meta on its crawlers, Microsoft on Bing and AI answers and Common Crawl on CCBot.

#Three nuances you need to know

Google-Extended is not a separate crawler but only a token in robots.txt. Google states explicitly that Google-Extended has no user agent string of its own in HTTP requests. Googlebot does the crawling, and the token determines what Google may do with that content. According to Google, the token does not affect inclusion in Google Search and is not a ranking signal. For AI Overviews and AI Mode, Googlebot is the control. Be aware, though, that Google-Extended also governs grounding in Gemini apps, so blocking it can reduce your visibility there.

Applebot-Extended works the same way. Apple states that Applebot-Extended doesn't crawl pages itself. According to Apple, pages that block this token remain discoverable through Spotlight, Siri and Safari.

Fetch bots such as ChatGPT-User, Perplexity-User and Meta-ExternalFetcher don't always follow robots.txt. OpenAI, Perplexity and Meta all indicate that robots.txt may not apply, because a user is making the request.

#How robots.txt decides which rule applies

In robots.txt, a crawler follows only the group that most specifically matches its own name, and falls back on the User-agent: * group only if it has no group of its own. This is set out in RFC 9309, the official robots.txt standard. Google confirms in its robots.txt documentation that specific groups and the * group are not combined.

This has a practical consequence that many people miss. Say you block the /admin/ folder under * and also create a separate group for GPTBot. GPTBot then ignores the * group completely, including the /admin/ rule. So repeat any rules that should apply to everyone in each separate group.

Within a group, the most specific rule wins, measured by the length of the path. Disallow: /blog/ therefore loses to Allow: /blog/guide/ for everything under /blog/guide/. If an Allow rule and a Disallow rule are equally specific, RFC 9309 says the Allow rule should win. Bot names are not case-sensitive: gptbot and GPTBot both work. Paths are.

#Example: visible in AI answers, but no training

With this file, search and fetch bots stay welcome while training bots and training tokens are excluded. It's a reasonable choice if you want reach in AI answers but don't want to make your content available for model training.

# Exclude training bots and training tokens
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

# All other bots, including OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, Googlebot, Applebot and bingbot
User-agent: *
Disallow: /admin/

Sitemap: https://www.yourcompany.com/sitemap.xml

Two caveats about this example. By blocking Google-Extended, you also exclude grounding in Gemini apps. If you do want to be visible there, leave that rule out. For meta-externalagent, Meta mentions indexing content for its own products as well as training. So it isn't a pure training bot.

#Example: allow everything

If you want maximum visibility and have no objection to training, a short file without separate AI groups is enough. Every bot then falls under the * group.

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.yourcompany.com/sitemap.xml

You don't need a separate group for each AI bot here. Only add one if you want that bot to behave differently from the rest. An llms.txt file complements robots.txt rather than replacing it: robots.txt controls access, while llms.txt gives language models a summary of your site.

#Microsoft controls AI use with meta tags

Microsoft doesn't use a separate AI token in robots.txt, but the NOCACHE and NOARCHIVE robots meta tags instead. According to Bing's announcement, NOARCHIVE excludes a page from chat answers and from training. With NOCACHE, Bing may only show the URL, the title and a snippet. Pages with either tag stay in the regular Bing search results. So don't block bingbot in robots.txt if you only want to limit AI use.

#The trap: your firewall blocks what robots.txt allows

An open robots.txt doesn't guarantee that AI bots can actually read your site, because a firewall or CDN can turn them away earlier in the chain. Robots.txt is a request to the bot. A firewall is a technical block that stops the request before your server sees it. The bot then gets an error code or a challenge page instead of your content.

Cloudflare is the best-known example. It offers AI Crawl Control, which lets you set allow or block rules per crawler, and the older Block AI bots setting. According to Cloudflare, from 15 September 2026 it applies new defaults to new domains. Under these, training and agent bots are blocked on pages with ads, while search bots remain allowed. Other hosting providers and security plugins also sometimes block unknown bots. None of this shows up in your robots.txt.

#How to test whether an AI bot actually reaches your site

You test this by requesting your site with an AI bot's user agent and checking what response you get. Flowproof does exactly this with the GPTBot user agent. You can do it yourself with one command in the terminal:

curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.yourcompany.com/

If you get status code 200, the firewall lets requests with that user agent through. A 403, 429 or 503 points to a block. Compare the result with a normal browser request: if that one does return 200, the block is tied to the user agent.

This test has a limit. Some firewalls also check the bot's IP address. OpenAI publishes its IP ranges for this purpose, for example for GPTBot. A test from your own IP address therefore mainly reveals user agent blocks. A block that only affects genuine OpenAI addresses, or only spoofed bots, won't show up this way. Your CDN or firewall logs will give you a definitive answer.

#Robots.txt is a request, not security

Robots.txt politely asks bots to skip certain parts of your site, but it doesn't technically stop anyone. RFC 9309 says these rules are not a form of access authorisation and no substitute for real security. Well-behaved bots from major companies respect it, but a malicious scraper can ignore the file. Common Crawl also warns about crawlers that falsely pose as CCBot.

The file is also public. Anyone can read it, so a Disallow: /secret/ actually tells people where something interesting is. Anything that really needs to stay private belongs behind a login or protected by server settings.

#How to check this on your own site

These steps let you check whether AI bots can read your site the way you want.

1. Open https://www.yourcompany.com/robots.txt and note which AI bots have their own group.
2. Check in each group that search bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot haven't been blocked by accident.
3. Check whether rules from the * group that should apply to everyone also appear in the separate groups.
4. Decide on your position on training and adjust the training bots and tokens accordingly.
5. Check the bot settings in your CDN, firewall or security plugin.
6. Run the curl test with the GPTBot user agent and compare it with a normal request.

Flowproof's free scan checks twelve AI bots in your robots.txt and runs the firewall test with the GPTBot user agent automatically. That way you can see in one overview whether your robots.txt and your firewall are saying the same thing.