AI crawler access
How robots.txt treats each crawler, and whether your server let our test fetches as AI bots through. One is allowed by robots.txt but turned away by your server, so check your firewall or CDN's bot settings.
View all 16 crawlers
| Crawler | Access | |
|---|---|---|
| Search and AI answersThese decide whether you can appear in results and answers. | ||
GooglebotGoogle | AllowedFollows the * rules | |
BingbotMicrosoft | AllowedFollows the * rules | |
OAI-SearchBotOpenAI | AllowedFollows the * rules, test fetch OK | |
Claude-SearchBotAnthropic | AllowedFollows the * rules, test fetch OK | |
PerplexityBotPerplexity | BlockedSee how to fix thisNamed in robots.txt | |
ApplebotApple | AllowedFollows the * rules | |
| Fetched when someone asksAn assistant reading your page because a person asked it to. | ||
ChatGPT-UserOpenAI | AllowedFollows the * rules, test fetch OK | |
Claude-UserAnthropic | AllowedFollows the * rules, test fetch OK | |
Perplexity-UserPerplexity | RefusedSee how to fix thisrobots.txt allows it, but your server returned 403 | |
| Model trainingBlocking these is a business choice, not a fault. | ||
GPTBotOpenAI | BlockedSee how to fix thisNamed in robots.txt | |
ClaudeBotAnthropic | BlockedSee how to fix thisNamed in robots.txt | |
Google-ExtendedGoogle | AllowedFollows the * rules | |
Applebot-ExtendedApple | AllowedFollows the * rules | |
meta-externalagentMeta | AllowedFollows the * rules | |
AmazonbotAmazon | AllowedFollows the * rules | |
CCBotCommon Crawl | BlockedSee how to fix thisNamed in robots.txt | |
View robots.txt
User-agent: *
Disallow: /basket
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: CCBot
Disallow: /
Sitemap: https://www.fernhillbakery.example/sitemap.xml