robots.txt returns a server error

CriticalCritical priority, in Robots.txt, checked once for the whole site

Short answer

robots.txt returns a server error means: when robots.txt fails with 5xx, Google stops crawling the whole site until it works again. To fix it: make robots.txt return 200 (or 404 if you have no rules).

By the SmooxSEO team. Updated on 16 November 2026.

What it means and why it matters

When robots.txt fails with 5xx, Google stops crawling the whole site until it works again.

How to fix it

Make robots.txt return 200 (or 404 if you have no rules).

Common causes

  • The robots.txt address is handled by the app and crashes, or the server is overloaded.
  • A firewall or CDN rule that answers errors on /robots.txt.

How to spot it

Open yourdomain.com/robots.txt: you should get the file (or a clean 404 if you have none), never a 5xx error. When robots.txt answers 5xx, Google treats the whole site as blocked until it can read the file again. SmooxSEO checks the robots.txt answer on every crawl.

Mistakes to avoid

  • Generating robots.txt with heavy app code: serve it as a simple static file.
  • Returning 5xx when you have no robots.txt: a missing file should answer 404, which means "no rules".

How serious is it?

The site, or its key pages, can drop out of Google or can't be reached. Fix it first.

About robots.txt checks

The robots.txt file tells crawlers which parts of the site they may visit. A wrong rule can hide a whole site from search engines.

Find this problem on your site

SmooxSEO checks your whole site for this problem and 311 others, lists the details, and tells you what to fix first.

Related robots.txt checks

All robots.txt checksAll 312 checks