How the biggest Dutch websites treat AI crawlers

#Key findings

• 15.5% of the sites studied block GPTBot, the bot OpenAI uses to collect training data. That is a legitimate choice: according to OpenAI it does not affect ChatGPT Search.
• 8% also block OAI-SearchBot, the bot ChatGPT uses to show sites in search results. A site that excludes this bot will not appear as a source in ChatGPT Search.
• 51 of the 65 OAI-SearchBot blocks (78.5%) are in a group of the bot's own. So it is usually a deliberate rule, not a general User-agent: * that hits AI bots by accident.
• 0.6% block Googlebot. Of the sites that keep out at least one AI search bot, 94.2% leave Googlebot open: they exclude AI search, but not regular search.
• 12.4% have an llms.txt. Which AI systems use that file is unconfirmed; we only measure whether it exists.

#Training bots and search bots are often blocked together

AI providers use separate bots for training models and for showing search results. OpenAI has GPTBot (training), OAI-SearchBot (search) and ChatGPT-User (pages a user asks it to fetch). Anthropic has ClaudeBot and Claude-SearchBot, and Google has Google-Extended alongside Googlebot.

Of the 126 sites that block GPTBot, 50.8% also block OAI-SearchBot. For 23.8%, OAI-SearchBot stays fully open: those sites keep out training but remain findable in ChatGPT Search. The rest are partly open.

If you only want to keep out training, the latter is the intended setup. A site that blocks both bots in one rule also removes itself as a source in ChatGPT Search. That can be a deliberate choice, for example for publishers with paid content, but it helps to know these are two separate decisions.

In total, 17.9% block at least one training bot and 10.6% block at least one AI search bot (OAI-SearchBot, PerplexityBot or Claude-SearchBot).

#Blocks via User-agent: *

A bot without a group of its own in robots.txt follows the User-agent: * group. If that group says Disallow: /, the site is closed to that bot, even if nobody meant it for AI bots.

For OAI-SearchBot this happens in 14 cases (21.5% of the blocks). In 9 of them, Googlebot is open through a group of its own: the site uses an allowlist of permitted search engines that does not include the AI search bots. On 5 sites everything is closed, Google included.

That allowlist is a common pattern that was set up before AI search engines existed. If you want ChatGPT, Perplexity or Claude to be able to show the site as a source, you have to add those bots to it separately.

#llms.txt

101 of the 814 sites (12.4%) have a file at /llms.txt that is served as text. We only counted files that are not an HTML page, so an error page with status 200 does not count. We did not assess the content. More about the file, and what is and is not known about it, is in the article on llms.txt.

#Method

• Date: 21 October 2026.
• List: the Tranco list 262W9 (averaged over 2026-09-21 – 2026-10-20), a research ranking of popular domains. We took the first 1,000 domains ending in .nl, in rank order.
• Requests: per domain, one request to https://<domain>/robots.txt (if there was no response, also https://www.<domain>/robots.txt) and one to /llms.txt on the same host. No other pages. User agent: FlowproofResearch/1.0 (+https://flowproof.nl/methode), at most 4 requests at a time, 10-second timeout, redirects followed.
• Rules: robots.txt read according to RFC 9309: a bot's own group takes precedence over User-agent: *, groups for the same bot are merged, the longest matching rule wins and Allow wins ties. Product tokens are case-insensitive.
• Status per bot: blocked if the homepage (/) is closed to the bot; partly if the homepage is open but at least one other path is disallowed that is not on our list of common private paths (such as /admin, /checkout and /account); otherwise open. The free AI crawler checker and the full scan use the same logic, described on the page about Flowproof's methodology.
• No robots.txt: a 404 or an HTML page instead of robots.txt counts as "no rules", so everything is open. That is what RFC 9309 prescribes. This applied to 104 sites; 710 had a robots.txt.
• Not counted: 186 domains returned no response, a server error (5xx) or a 401, 403 or 429. Many of them are technical domains without a website. A 403 or 429 is often a firewall response to our user agent and does not say what an AI bot gets, so we do not count it as "everything open". All percentages refer to the remaining 814 domains.

#Limitations

• robots.txt is a request, not a lock. A site that allows AI bots in robots.txt can still keep them out with a firewall or CDN rule. We did not measure that: we made no page requests as an AI bot.
• Homepage only. The status refers to the path /. A site can allow a bot on the homepage and keep it out of other sections; that falls under partly.
• Partly is a broad category. Disallowing filters, search results or internal paths that are not on our list also counts as partly. That is usually harmless, which is why the key findings only use blocked.
• Domains, not companies. The Tranco list also contains technical and redirecting domains. We do not name individual sites; this is about totals.
• Snapshot. robots.txt files change often. The figures apply to 21 October 2026.
• 12 bots. We studied the bots that Flowproof's scan also checks. Other AI bots are out of scope.

#Check it yourself

With the free AI crawler checker you can see for your own site which of these 12 bots robots.txt allows, partly allows or blocks, and through which rule. Questions about the study or the method: email jelle@peelmedia.nl.