AI crawler access
Are AI bots blocked by robots.txt or a firewall?
- ✕
PerplexityBot (Perplexity)
Blocked via its own rule. This bot indexes for Perplexity answers.
Add to robots.txt: User-agent: PerplexityBot Allow: / Without access, Perplexity can't cite you as a source.
- ✓
robots.txt present
robots.txt is reachable and readable.
- ✓
OAI-SearchBot (OpenAI)
Allowed — shows your site in ChatGPT search results.
- ✓
ChatGPT-User (OpenAI)
Allowed — fetches pages when a user asks for them.
- ✓
GPTBot (OpenAI)
Allowed — collects content to train models.
- ✓
Googlebot (Google)
Allowed — powers Google Search and AI Overviews.
- ✓
Google-Extended (Google)
Allowed — controls use in Gemini.
- ✓
Bingbot (Microsoft)
Allowed — powers Bing and Copilot.
- ✓
Claude-SearchBot (Anthropic)
Allowed — indexes for search in Claude.
- ✓
ClaudeBot (Anthropic)
Allowed — collects content for Claude.
- ✓
Applebot-Extended (Apple)
Allowed — controls use in Apple Intelligence.
- ✓
meta-externalagent (Meta)
Allowed — collects content for Meta AI.
- ✓
CCBot (Common Crawl)
Allowed — open dataset used by many AI models.
- ✓
Server allows AI bots
A request as OAI-SearchBot (ChatGPT's search bot) got status 200. No firewall blocking was found.
- ✓
No noai directive
No noai directive was found.