AI crawler access
How robots.txt treats each crawler, and whether your server let our test fetches as AI bots through.
View all 16 crawlers
| Crawler | Access | |
|---|---|---|
| Search and AI answersThese decide whether you can appear in results and answers. | ||
GooglebotGoogle | AllowedFollows the * rules | |
BingbotMicrosoft | AllowedFollows the * rules | |
OAI-SearchBotOpenAI | AllowedFollows the * rules, test fetch OK | |
Claude-SearchBotAnthropic | AllowedFollows the * rules, test fetch OK | |
PerplexityBotPerplexity | AllowedFollows the * rules, test fetch OK | |
ApplebotApple | AllowedFollows the * rules | |
| Fetched when someone asksAn assistant reading your page because a person asked it to. | ||
ChatGPT-UserOpenAI | AllowedFollows the * rules, test fetch OK | |
Claude-UserAnthropic | AllowedFollows the * rules, test fetch OK | |
Perplexity-UserPerplexity | AllowedFollows the * rules, test fetch OK | |
| Model trainingBlocking these is a business choice, not a fault. | ||
GPTBotOpenAI | AllowedFollows the * rules | |
ClaudeBotAnthropic | AllowedFollows the * rules | |
Google-ExtendedGoogle | AllowedFollows the * rules | |
Applebot-ExtendedApple | AllowedFollows the * rules | |
meta-externalagentMeta | AllowedFollows the * rules | |
AmazonbotAmazon | AllowedFollows the * rules | |
CCBotCommon Crawl | AllowedFollows the * rules | |
View robots.txt
User-Agent: *
Disallow: /try/
Disallow: /admin
# Prospect previews are unlisted and built for one named operator. They also carry
# `noindex, nofollow` in the page head — this is belt and braces.
#
# /admin is password-gated and sends `X-Robots-Tag: noindex` on every response. This line is
# only so a crawler does not waste requests on it — it is not what keeps it private.
Sitemap: https://www.parkalto.com/sitemap.xml