Free Tool

Which AI crawlers can
read your site?

Paste your robots.txt below and see, bot by bot, who you let in. Three kinds of crawler ask for your pages, and blocking each costs something different.

Three Kinds of Crawler
Training

Gathers pages to train future models. Blocking it keeps your content out of training corpora, but it does not remove you from AI answers.

Search

Builds the index that AI search answers cite. Blocking it removes you from those answers.

Assistant

Fetches your page live, at the moment a user asks. Blocking it means the assistant cannot read you in real time.

Check Your robots.txt

Verdicts follow robots.txt group matching for the site root: a crawler obeys its most specific user-agent group, or the * group if none names it. robots.txt is a request, not a wall — the major vendors honour it.

The Directory

The crawlers below are the ones that matter for AI visibility. This is the same classification table our own traffic monitoring runs on, kept current because we read its output every day.

User agentVendorKind
ChatGPT-UserOpenAIAssistant
OAI-SearchBotOpenAISearch
GPTBotOpenAITraining
Claude-UserAnthropicAssistant
Claude-SearchBotAnthropicSearch
ClaudeBotAnthropicTraining
anthropic-aiAnthropicTraining
Perplexity-UserPerplexityAssistant
PerplexityBotPerplexitySearch
MistralAI-UserMistralAssistant
DuckAssistBotDuckDuckGoAssistant
Google-ExtendedGoogleTraining
GoogleOtherGoogleTraining
Googlebot · also classic web search, blocking it costs ordinary search traffic tooGoogleSearch
bingbot · also classic web search, blocking it costs ordinary search traffic tooMicrosoftSearch
Applebot-ExtendedAppleTraining
ApplebotAppleSearch
AmazonbotAmazonTraining
meta-externalagentMetaTraining
FacebookBotMetaTraining
BytespiderByteDanceTraining
CCBotCommon CrawlTraining
cohere-aiCohereTraining
YouBotYou.comSearch
DiffbotDiffbotTraining
TimpibotTimpiTraining
omgiliWebz.ioTraining
ImagesiftBotImagesiftTraining
AgentTrustBotAgentTrustSearch
Common Questions
It depends what you sell. Blocking GPTBot keeps your pages out of OpenAI's training corpora; it does not remove you from ChatGPT’s answers, which come through OAI-SearchBot and ChatGPT-User. If being found when buyers ask AI matters to you, block training bots if you wish, but think hard before blocking search and assistant bots.
One vendor, three jobs. GPTBot collects training data, OAI-SearchBot builds the search index ChatGPT cites, and ChatGPT-User fetches a page live when a user asks about it. Anthropic runs the same trio (ClaudeBot, Claude-SearchBot, Claude-User). robots.txt lets you treat each differently, that is the point of this page.
From operation, not from a blog post. It is the classification table behind our own traffic monitoring, every crawler on it has been seen and verified in live server logs, and the list grows when an unrecognised agent shows up there.
It is a published request and compliance is voluntary, but OpenAI, Anthropic, Google, Apple and the other major vendors document and honour it. What it will not stop is a scraper that never announces itself; that is a firewall problem, not a robots.txt one.