AI crawler access
How robots.txt treats each crawler, and whether your server let our test fetches as AI bots through.
View all 16 crawlers
| Crawler | Access | |
|---|---|---|
| Search and AI answersThese decide whether you can appear in results and answers. | ||
GooglebotGoogle | AllowedFollows the * rules | |
BingbotMicrosoft | AllowedFollows the * rules | |
OAI-SearchBotOpenAI | AllowedFollows the * rules, test fetch OK | |
Claude-SearchBotAnthropic | AllowedFollows the * rules, test fetch OK | |
PerplexityBotPerplexity | AllowedFollows the * rules, test fetch OK | |
ApplebotApple | AllowedFollows the * rules | |
| Fetched when someone asksAn assistant reading your page because a person asked it to. | ||
ChatGPT-UserOpenAI | AllowedFollows the * rules, test fetch OK | |
Claude-UserAnthropic | AllowedFollows the * rules, test fetch OK | |
Perplexity-UserPerplexity | AllowedFollows the * rules, test fetch OK | |
| Model trainingBlocking these is a business choice, not a fault. | ||
GPTBotOpenAI | AllowedFollows the * rules | |
ClaudeBotAnthropic | AllowedFollows the * rules | |
Google-ExtendedGoogle | AllowedFollows the * rules | |
Applebot-ExtendedApple | AllowedFollows the * rules | |
meta-externalagentMeta | AllowedFollows the * rules | |
AmazonbotAmazon | AllowedFollows the * rules | |
CCBotCommon Crawl | AllowedFollows the * rules | |
View robots.txt
User-Agent: *
Allow: /
Disallow: /admin
Disallow: /api/
Sitemap: https://crawlable.review/sitemap.xml