Free SEO tool

Robots.txt and AI crawler checker

Paste your robots.txt file below to check homepage access for common search and AI crawlers. The checker analyzes your pasted rules locally in your browser.

Robots.txt is a request to compliant crawlers, not an access control. Firewall and CDN rules can still block crawlers. Google-Extended uses robots.txt rules but does not fetch pages with its own user agent.

Why check AI crawlers?

Search crawlers can help content appear in answers, while training crawlers collect data for model development. Decide which access you want to allow. Rules for one bot do not necessarily apply to another.

Frequently asked questions

Does this check my live website?

No. It evaluates the text you paste for the homepage only. Check the file at /robots.txt on your site to make sure it matches.

Does blocking a crawler remove my page from search?

Not necessarily. A blocked URL can still be indexed from links. To request removal from search, use noindex on an accessible page and consult the search engine’s documentation.

Related tools and guides