AI robots stopped by a firewall

MediumMedium priority, in AI search, checked once for the whole site

Short answer

AI robots stopped by a firewall means: asked with their real names, these robots got an error or a challenge page while a normal browser got the site. Often Cloudflare's "Block AI bots" setting or a bot-protection rule. Tested from Smoox's server: real robots come from their own addresses, so treat it as a strong hint. To fix it: in Cloudflare (Security > Bots) or your firewall, allow the AI search robots you want to be quoted by. Training robots can stay blocked.

By the SmooxSEO team. Updated on 16 November 2026.

What it means and why it matters

Asked with their real names, these robots got an error or a challenge page while a normal browser got the site. Often Cloudflare's "Block AI bots" setting or a bot-protection rule. Tested from Smoox's server: real robots come from their own addresses, so treat it as a strong hint.

How to fix it

In Cloudflare (Security > Bots) or your firewall, allow the AI search robots you want to be quoted by. Training robots can stay blocked.

Common causes

  • A CDN or firewall "bot fight" mode that blocks or challenges unknown user agents.
  • Rate limits that stop crawlers after a few requests.
  • A CAPTCHA or JavaScript challenge shown to every non-browser visitor.

How to spot it

robots.txt may allow AI crawlers while your firewall still blocks them. Check your CDN or firewall settings for bot protection, and its logs for blocked AI user agents. SmooxSEO sends real requests with each AI crawler's user agent and compares the answers with a normal browser's.

Mistakes to avoid

  • Allowing the bots in robots.txt and assuming they get in: the firewall decides first.
  • Allowing user agents by name only: some firewalls also verify the crawler's IP ranges, which the AI providers publish.

How serious is it?

The page still ranks, but less well, or a real opportunity is lost. Fix it when you can.

About ai search checks

Whether AI assistants (ChatGPT, Claude, Perplexity, Gemini, Copilot) can read and quote your site: AI crawler access, llms.txt and answer-ready content.

Find this problem on your site

SmooxSEO checks your whole site for this problem and 311 others, lists the details, and tells you what to fix first.

Related ai search checks

All ai search checksAll 312 checks