Is Cloudflare (or your website host) hiding you from ChatGPT? How to check and fix it

The short answer

It can. Cloudflare and some hosts can block or challenge AI crawlers even when your robots.txt allows them: through AI access policies, Bot Fight Mode, or a “block AI” switch that also catches search bots. Check with a free crawler test, then allow the search and agent bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) and, if you want, keep only the training bots out.

Search bots, agent bots and training bots: which ones matter

AI companies send different crawlers for different jobs. For being recommended, only some of them matter, and blocking the wrong one is how most businesses disappear by accident:

AI crawlers by job, and what blocking them does
Company	Search (finds you for answers)	Agent (opens a page when someone asks)	Training (learns from your pages)
OpenAI (ChatGPT)	`OAI-SearchBot`: blocked sites don’t appear in ChatGPT search answers	`ChatGPT-User`: robots.txt may not apply	`GPTBot`: training only, no effect on answers
Anthropic (Claude)	`Claude-SearchBot`: blocking may reduce visibility	`Claude-User`	`ClaudeBot`
Perplexity	`PerplexityBot`: “surface and link websites” in results	`Perplexity-User`: generally ignores robots.txt	No separate training bot

The rule for a local business: allow every search and agent bot. Blocking training bots is your choice and doesn’t remove you from today’s answers. The full list of AI crawlers.

How Cloudflare can hide you from ChatGPT

Cloudflare protects a large share of small-business websites, often switched on by the web designer, and it has several settings that can turn AI crawlers away. Your robots.txt can say “welcome” while Cloudflare still blocks the door. These are the four to know.

AI Crawler and Agent access policies. Since July 2026 Cloudflare sorts AI bots into three groups (Search, Agent and Training) instead of one “Block AI bots” switch, on every plan including Free. Each group can be allowed, blocked everywhere, or blocked only on pages with ads. Cloudflare’s announcements describe stricter presets for domains added since 15 September 2026, while its documentation (November 2026) says all three are allowed by default: check yours, don’t assume.
“Block” on Training now also stops Google and Bing. Since 15 September 2026, Block applies to mixed-use crawlers such as Googlebot, Bingbot and Applebot, which crawl for search and for AI training. Cloudflare says “Selecting Block stops them entirely.” To keep training out but stay in search, choose “Disallow AI Training” instead.
Bot Fight Mode (Free plan). It answers suspected bots with JavaScript challenges, and Cloudflare says it can’t be bypassed with WAF custom rules or Page Rules. AI crawlers don’t run JavaScript the way a browser does, so a challenge stops them. Perplexity’s own documentation warns that Bot Fight Mode “can still challenge Perplexity bots.”
Super Bot Fight Mode and security levels (paid plans). More control, including an option to allow verified bots, but “definitely automated” traffic set to block or challenge, or “Under Attack” mode, can stop AI crawlers too.

Cloudflare’s managed robots.txt is the exception: it adds rules for training bots only (GPTBot, ClaudeBot, CCBot, Google-Extended and a few others) and leaves OAI-SearchBot, Claude-SearchBot and PerplexityBot alone.

Do other hosts and website builders block AI crawlers?

Most don’t by default, but several have a switch, and some switches catch search or agent bots too:

AI bot blocking at hosts, CDNs and website builders (checked November 2026)
Platform	Default	Where	Watch out
Squarespace	Off	Settings › Crawlers › Block known AI crawlers	Blocks 26 bots, mostly training; DuckAssistBot is on the list
Wix	No blocking	SEO & GEO › Robots.txt Editor	Wix’s suggested list includes ChatGPT-User, which ChatGPT uses to open pages
WordPress.com	Off	Settings › Reading › Prevent third-party sharing	Adds known AI bots to robots.txt
SiteGround	Training bots blocked	Ask support (Help Desk › AI Crawlers Setup)	Bots for real users are allowed; its anti-bot firewall can still challenge some
Hostinger	No blocking	robots.txt in File Manager	Hostinger says its CDN security “can block crawlers even when robots.txt allows them”
Vercel	Off	Firewall › Bot Management › AI Bots ruleset	“AI Assistants” is a separate group from “AI Crawlers”: keep it allowed
Netlify	No blocking	User Agent Blocker extension (opt-in)	A preset list; check which bots it names
Akamai, Fastly	No stated default	Bot management policies	Both separate training, search and fetcher bots: allow the last two

How common is this?

More common than owners think, and it usually happens without anyone deciding to block ChatGPT. When we ran our AI Visibility Index on 100 local business websites, one of them turned our checker away entirely, before we could ask AI a single question about it.

49%

From our dataof the top 100 US and UK news sites blocked OpenAI’s search crawler, OAI-SearchBot, and 67% blocked PerplexityBot. Many meant to block training and blocked search too.BuzzStream: which bots publishers block (Apr 2026)

Cloudflare’s own data shows the same direction: AI crawlers were the most often fully blocked bots in robots.txt files in 2025, with GPTBot first. Training bots are a fair thing to block. Search bots are where a local business loses customers.

How to check if your site blocks AI crawlers

Run the free AI crawler checker. The checker reads your robots.txt, loads your homepage as OAI-SearchBot, Claude-SearchBot, PerplexityBot and others, and flags a 403 or a challenge page.
Open yoursite.com/robots.txt. On Cloudflare, managed rules are added on top of your own file, so the file visitors get can differ from the one on your server. Look for Disallow rules under the search and agent bots.
On Cloudflare, look at the evidence. AI Crawl Control › Crawlers shows each AI crawler, its requests and whether it’s allowed or blocked. Security › Analytics › Events, filtered by user agent, shows challenges and blocks (sampled on Free, up to 31 days).
Check your server logs, if you have them. Compare visits from OAI-SearchBot, Claude-SearchBot and PerplexityBot with the IP ranges each company publishes. Real crawler visits that get 403 or 503 are blocked.

One limit of every test from outside: a request that only says it’s OAI-SearchBot isn’t the real bot, and firewalls can treat the real one differently, in either direction. Cloudflare’s logs and your server logs are the final word.

How to fix it on Cloudflare, step by step

Set the AI access policies. Security › Settings › filter “Bot traffic” › AI Crawler and Agent access policies. Search: Allow. Agents: Allow (Cloudflare recommends this for personal agents). Training: your choice, and if you limit it, use Disallow AI Training, not Block.
Allow the search bots in AI Crawl Control. AI Crawl Control › Crawlers: make sure OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User say Allow.
Check Bot Fight Mode. Security › Settings › Bot traffic › Bot fight mode. If your Security events show it challenging AI search bots, turn it off: on the Free plan it can’t be told to skip them.
On a paid plan, allow verified bots. In Super Bot Fight Mode, set Verified bots to Allow, and don’t block or challenge “Definitely automated” traffic without checking who it catches.
Test again. OpenAI says robots.txt changes take about 24 hours to reach its search crawler. Run the checker again the next day, and check your Security events.

A robots.txt that lets AI find you

This allows the bots that put you in AI answers and, optionally, keeps your pages out of AI training. Bots you don’t name are allowed by default, so the first block mostly makes your intent clear:

robots.txt

# Let AI search and answer bots read the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Optional: keep your pages out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://www.yourbusiness.com/sitemap.xml

Keep it at the root of your domain, and remember robots.txt is a request, not a lock: your host’s firewall still decides who actually gets in.

Myths about blocking AI crawlers

“Blocking GPTBot removes me from ChatGPT.” No: GPTBot is for training. ChatGPT search uses OAI-SearchBot.
“My robots.txt allows everything, so I’m fine.” A firewall, bot protection or CDN setting can still block crawlers that robots.txt welcomes.
“Blocking AI bots is all or nothing.” Cloudflare, Vercel, Akamai and Fastly all separate training bots from search and agent bots. Block the first, keep the others.
“Blocking Google-Extended hurts my Google ranking.” Google says it doesn’t affect inclusion or ranking in Search. It’s about Gemini training.

Questions

Does Cloudflare block ChatGPT by default?

It depends on when your domain joined and what you chose. Cloudflare announced stricter AI defaults for new domains in 2025 and 2026, while its current documentation says new domains allow Search, Agent and Training bots by default. Existing domains keep their settings. Open Security, Settings, Bot traffic, AI Crawler and Agent access policies to see yours.

Does Bot Fight Mode block AI crawlers?

It can. Bot Fight Mode challenges suspected bots with JavaScript, which crawlers don't run, and Cloudflare says it can't be bypassed with WAF custom rules. If your security events show it challenging OAI-SearchBot or PerplexityBot, turn it off, or on a paid plan use Super Bot Fight Mode with verified bots allowed.

Should I block GPTBot?

It's your choice. GPTBot collects pages for training OpenAI's models; blocking it doesn't remove you from ChatGPT search, which uses OAI-SearchBot. Keep OAI-SearchBot and ChatGPT-User allowed if you want to be recommended.

How long until ChatGPT can read my site after I fix it?

OpenAI says robots.txt changes reach its search crawler in about 24 hours. Firewall changes on Cloudflare apply at once. Being recommended takes longer: the crawler has to visit again, and answers change over weeks.

I'm not on Cloudflare. Can my host still block AI bots?

Yes. Some hosts block training bots at the server, security firewalls can challenge crawlers, and website builders such as Squarespace, Wix and WordPress.com have AI blocking options. Run a crawler check to see what actually happens.

Sources

Narek Nikoghosyan · Founder and CEO
Narek Nikoghosyan is the founder and CEO of stuckUP, the AI visibility checker for local businesses.