Page hidden from some AI robots

MediumMedium priority, in AI search

Short answer

Page hidden from some AI robots means: a rule for specific robots (<meta name="OAI-SearchBot" content="noindex"> or an X-Robots-Tag header naming them) keeps this page out of those assistants' answers. To fix it: remove the rule for the answer robots listed, unless you really want this page left out of ChatGPT, Claude or Perplexity answers. Rules for training robots (GPTBot, ClaudeBot...) can stay.

By the SmooxSEO team. Updated on 16 November 2026.

What it means and why it matters

A rule for specific robots (<meta name="OAI-SearchBot" content="noindex"> or an X-Robots-Tag header naming them) keeps this page out of those assistants' answers.

How to fix it

Remove the rule for the answer robots listed, unless you really want this page left out of ChatGPT, Claude or Perplexity answers. Rules for training robots (GPTBot, ClaudeBot...) can stay.

Common causes

  • Page-level rules (meta robots or X-Robots-Tag) aimed at specific AI bots.
  • robots.txt rules that block some folders for some AI crawlers only.

How to spot it

Look for meta tags like <meta name="GPTBot" content="noindex"> or nosnippet, and robots.txt rules for specific AI user agents on specific folders. SmooxSEO reports which AI crawlers can't read which pages.

Mistakes to avoid

  • Applying a rule site-wide that was meant for one private section.

How serious is it?

The page still ranks, but less well, or a real opportunity is lost. Fix it when you can.

About ai search checks

Whether AI assistants (ChatGPT, Claude, Perplexity, Gemini, Copilot) can read and quote your site: AI crawler access, llms.txt and answer-ready content.

Find this problem on your site

SmooxSEO checks every page of your site for this problem and 311 others, lists the pages concerned, and tells you what to fix first.

Related ai search checks

All ai search checksAll 312 checks