The short answer
AI crawlers come in two kinds. Search crawlers like OAI-SearchBot (ChatGPT), Claude-SearchBot and PerplexityBot fetch pages to answer questions, so blocking them can remove you from AI answers. Training crawlers like GPTBot and ClaudeBot collect pages to train AI models, and blocking them is your choice. Most local businesses should allow every search crawler.
Two kinds of AI crawlers
Every big AI company now sends more than one crawler, and they do different jobs. The difference matters, because blocking the wrong one can quietly remove you from AI answers:
- Search and answer crawlers fetch pages so the assistant can answer a question right now: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot and Perplexity-User (Perplexity). Block these and your site can't be used in their answers.
- Training crawlers collect pages that may be used to train future AI models: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's Gemini), Applebot-Extended (Apple) and CCBot (Common Crawl, a public dataset many AI models learn from). Blocking these is a choice about your content, and it doesn't remove you from today's AI answers.
The AI crawlers you'll see in your logs
| Crawler | Company | What it does | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | Finds pages for ChatGPT search | Your site can drop out of ChatGPT search results |
ChatGPT-User | OpenAI | Opens a page when a ChatGPT user asks it to | Often ignored: OpenAI says robots.txt rules may not apply |
GPTBot | OpenAI | Collects pages for training models | Not used for training; no effect on ChatGPT search |
Claude-SearchBot | Anthropic | Improves the results of Claude's web search | Less visible in Claude's search answers |
Claude-User | Anthropic | Opens pages when Claude users ask questions | Claude can't read your site for its users |
ClaudeBot | Anthropic | Collects pages for training models | Your future pages are left out of training |
PerplexityBot | Perplexity | Indexes pages for Perplexity's answers | Perplexity says to allow it to appear in results |
Perplexity-User | Perplexity | Opens pages for a user's question | Not used for crawling or training |
Google-Extended | Training and grounding for Gemini | No effect on Google Search or AI Overviews | |
Googlebot | Google Search, AI Overviews and AI Mode | You disappear from Google, AI features included | |
Bingbot | Microsoft | Bing search, which Copilot answers from | You disappear from Bing and Copilot |
Applebot-Extended | Apple | Not a crawler: decides whether pages Applebot reads may train Apple's AI | No effect on Apple's search results |
CCBot | Common Crawl | A public web archive many AI models are trained on | Left out of future Common Crawl archives |
ClaudeBot: what it is, and is it safe to block?
ClaudeBot is Anthropic's training crawler. Anthropic says it collects public web content that may help train its Claude models. Blocking it keeps your future pages out of that training, and it's a legitimate choice.
What blocking ClaudeBot doesn't do is stop Claude from answering questions with your site: that's Claude-SearchBot and Claude-User. If you block ClaudeBot, leave those two allowed. Anthropic also notes it doesn't publish its IP addresses, so blocking by IP isn't a reliable opt-out; use robots.txt.
GPTBot vs OAI-SearchBot
This is the mix-up we see most. GPTBot collects pages that may be used to train OpenAI's models. OAI-SearchBot is the crawler OpenAI uses to show websites in ChatGPT search. They're independent: you can block GPTBot and still be found in ChatGPT, but if you block OAI-SearchBot you won't be.
OpenAI says it takes about 24 hours for a robots.txt change to reach its systems.
Which crawlers should a local business allow?
Our advice for a restaurant, clinic, hotel or law firm that wants customers to find it through AI:
- Allow every search and answer crawler. OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot and Bingbot.
- Decide about training crawlers on purpose. Allowing them helps future AI models know your business; blocking them protects your content from training. Either is fine. Just don't block search crawlers by accident while doing it.
- Check your firewall or CDN, not only robots.txt. Bot protection can turn AI crawlers away even when robots.txt allows them (see below).
User-agent: *
Allow: /User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /Firewalls and CDNs: the block you can't see in robots.txt
Many sites block AI crawlers without anyone deciding to. Firewalls and CDNs such as Cloudflare have “bot protection” and AI-crawler settings that answer crawlers with a “403 Forbidden” or a challenge page, while the site still works fine in a browser. robots.txt says “come in”, but the door is locked.
The only way to know is to load your pages the way each crawler does. If one is blocked, look for the bot or AI crawler settings in your firewall or CDN dashboard and allow verified AI search bots there.
Being readable is step one
Letting AI crawlers in only means AI can use your site. Whether it recommends you depends on what it finds: how clearly your site says what you do and where, and whether the guides and directories it trusts mention you. Our plain-English guide to GEO covers the rest.
Questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot only collects pages for training. ChatGPT search uses OAI-SearchBot, so as long as OAI-SearchBot is allowed, ChatGPT can still find and cite your site.
Is ClaudeBot safe to block?
Yes, if you don't want your pages used for training Anthropic's models. Keep Claude-SearchBot and Claude-User allowed, or Claude may not be able to use your site when people ask it questions.
Does blocking Google-Extended affect Google Search or AI Overviews?
No. Google says Google-Extended doesn't affect a site's inclusion or ranking in Google Search. AI Overviews and AI Mode use Googlebot and the normal search index.
How long until a robots.txt change takes effect?
OpenAI says about 24 hours for its search crawler; Perplexity says up to 24 hours. Other crawlers pick up changes the next time they read your robots.txt.
Do AI crawlers follow robots.txt?
The automatic crawlers from OpenAI, Anthropic, Google and Perplexity say they do. Fetchers that open a page because a user asked, like ChatGPT-User, may not follow robots.txt.
Sources
Narek Nikoghosyan · Founder and CEO
Narek Nikoghosyan is the founder and CEO of stuckUP, the AI visibility checker for local businesses.