How to monitor robots.txt changes

  • seo
  • robots

robots.txt is the one file that can remove a site from search without breaking anything a visitor would notice. A staging copy with Disallow: / ships to production, a CMS plugin rewrites the file, a firewall starts refusing crawlers. The homepage keeps answering 200, so uptime checks stay green. Monitoring robots.txt means reading it the way a crawler does, on a schedule, and comparing what it says now with what it said before.

How do I check robots.txt right now?

Fetch it with the status code and headers, so you see what a crawler sees:

curl -sS -i https://example.com/robots.txt

Read three things in the output:

  • The status code. 200 means the rules below apply. 404 or 410 means there are no rules, so everything is allowed. A 5xx is the dangerous one: per Google's robots.txt documentation, Googlebot stops crawling the whole site for the first 12 hours of 5xx answers, then falls back to the last copy it cached.
  • Redirects. Google follows at least five redirect hops for robots.txt. A redirect to a login page or a different host is a common accident after a migration.
  • The rules. Look for Disallow: / under User-agent: * or under User-agent: Googlebot.

How do I tell whether a rule blocks a page?

Crawlers do not read the file top to bottom. Under RFC 9309, the standard Google and Bing implement:

  1. A crawler uses the group that names it, for example User-agent: Googlebot. Only if no group names it does it use User-agent: *. A Googlebot group therefore replaces the * rules for Googlebot, it does not add to them.
  2. Inside that group, the longest matching rule wins. Allow: /shop/ beats Disallow: /shop for /shop/shoes.
  3. When an Allow and a Disallow match with the same length, Allow wins.
  4. * matches any run of characters, and $ at the end anchors the rule to the end of the URL path.

Google Search Console's robots.txt report shows the last version Google fetched and any parsing errors, which is the quickest way to see what Google itself believes.

Why isn't checking it once enough?

Because the file changes when something else changes: a deploy, a CMS update, a new firewall rule. A check you run by hand describes the file on the day you ran it. The changes that hurt are the ones nobody meant to make, which means nobody thinks to check.

A cron job that fetches the file, diffs it against yesterday's copy, and emails you on a difference covers the edits. It does not cover the box running the cron going down, and a raw diff alerts on a comment change as loudly as on Disallow: /.

What should automatic robots.txt monitoring watch?

Four things, because they fail differently:

  1. Reachability. A 5xx, a timeout, or a redirect loop on robots.txt pauses crawling. That is an outage, and it should page like one.
  2. Your critical paths. Decide which paths must stay crawlable, at least /, maybe /shop or /blog, and alert the moment any of them is disallowed for the crawlers you care about.
  3. New Disallow rules, with a diff, even when no critical path is blocked yet. A new rule is usually the first sign of a bad deploy.
  4. The file disappearing. A missing robots.txt allows everything, so it is not an outage, but deliberate blocks and Sitemap: lines vanish with it.

A 401 or 403 on robots.txt deserves its own label. It is almost always a firewall refusing the monitoring tool rather than a real rule, and treating it as "the rules changed" sends you looking in the wrong place.

YoPingMe's robots.txt monitor covers all four from your check regions and confirms a change on two checks in a row before it alerts, so one odd response never pages anyone. It is part of the Pro and Team plans. Its companion, sitemap monitoring, watches the other half of crawl health: how many URLs your sitemap lists.