The robots.txt file tells crawlers which parts of the site they may visit. A wrong rule can hide a whole site from search engines.
SmooxSEO runs 5 robots.txt checks on every page of your site:
The site, or its key pages, can drop out of Google or can't be reached. Fix it first.
"Disallow: /" stops Google from crawling every page. Often left over from development.
When robots.txt fails with 5xx, Google stops crawling the whole site until it works again.
The page still ranks, but less well, or a real opportunity is lost. Fix it when you can.
Google needs the site's CSS and JavaScript to render pages; robots.txt blocks some of them.
Lines search engines can't understand are ignored, so some rules don't apply.
Good practice with a small effect, or something that is often intentional. Nice to have.
Without robots.txt everything may be crawled; that is fine, but you have no control and no place to list your sitemap.