AI search robots blocked
Short answer
AI search robots blocked means: robots.txt stops the robots that ChatGPT, Claude, Perplexity and others use to find answers. Your pages can't be quoted or linked in those answers. To fix it: allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and similar robots in robots.txt (you can still block the training robots).
By the SmooxSEO team. Updated on 16 November 2026.
What it means and why it matters
robots.txt stops the robots that ChatGPT, Claude, Perplexity and others use to find answers. Your pages can't be quoted or linked in those answers.
How to fix it
Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and similar robots in robots.txt (you can still block the training robots).
Common causes
- A robots.txt rule that blocks AI crawlers along with the ones you meant to block.
- A "block AI" setting in a CMS, CDN or security tool that also blocks search assistants.
- Copying a robots.txt from another site without reading it.
How to spot it
Open yourdomain.com/robots.txt and look for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Blocking the training crawlers (GPTBot, ClaudeBot, CCBot) is a choice; blocking these search and answer crawlers keeps you out of AI answers. SmooxSEO tests each AI crawler against your robots.txt and with live requests.
Example
User-agent: OAI-SearchBot
Disallow: /# Allow AI search, block AI training if you prefer
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Mistakes to avoid
- Blocking all AI bots to stop model training: you also disappear from ChatGPT search, Claude and Perplexity answers.
- Allowing the bots in robots.txt while a firewall still blocks them.
How to fix it on your platform
- robots.txt
- Remove Disallow rules for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. You can still block training crawlers such as GPTBot and ClaudeBot.
- Cloudflare
- Check Security > Bots and the AI crawler settings: allow the search and user-request bots of the assistants you want to appear in.
Questions people ask
Can I block AI training but still appear in ChatGPT answers?
Yes. OpenAI uses GPTBot for training and OAI-SearchBot for search; Anthropic uses ClaudeBot for training and Claude-SearchBot for search. Block the training crawler and allow the search one.
Does blocking AI bots affect Google?
Blocking these bots doesn't change Google rankings. Google uses Googlebot, and the separate Google-Extended token only controls use of your content to train Gemini.
How serious is it?
A page can't be found, indexed or trusted, or search engines can't read part of the site. Fix it soon: it costs traffic.
About ai search checks
Whether AI assistants (ChatGPT, Claude, Perplexity, Gemini, Copilot) can read and quote your site: AI crawler access, llms.txt and answer-ready content.
Find this problem on your site
SmooxSEO checks your whole site for this problem and 311 others, lists the details, and tells you what to fix first.