# How to monitor robots.txt changes

> One Disallow line can drop a site out of search while every page still loads. How to check robots.txt by hand, and what watching it automatically has to cover.

Published: 2026-11-12 | Canonical: https://yoping.me/guides/how-to-monitor-robots-txt-changes

robots.txt is the one file that can remove a site from search without
breaking anything a visitor would notice. A staging copy with
`Disallow: /` ships to production, a CMS plugin rewrites the file, a
firewall starts refusing crawlers. The homepage keeps answering 200, so
uptime checks stay green. Monitoring robots.txt means reading it the way
a crawler does, on a schedule, and comparing what it says now with what
it said before.

## How do I check robots.txt right now?

Fetch it with the status code and headers, so you see what a crawler
sees:

```
curl -sS -i https://example.com/robots.txt
```

Read three things in the output:

- **The status code.** 200 means the rules below apply. 404 or 410
  means there are no rules, so everything is allowed. A 5xx is the
  dangerous one: per Google's robots.txt documentation, Googlebot stops
  crawling the whole site for the first 12 hours of 5xx answers, then
  falls back to the last copy it cached.
- **Redirects.** Google follows at least five redirect hops for
  robots.txt. A redirect to a login page or a different host is a
  common accident after a migration.
- **The rules.** Look for `Disallow: /` under `User-agent: *` or under
  `User-agent: Googlebot`.

## How do I tell whether a rule blocks a page?

Crawlers do not read the file top to bottom. Under RFC 9309, the
standard Google and Bing implement:

1. A crawler uses the group that names it, for example
   `User-agent: Googlebot`. Only if no group names it does it use
   `User-agent: *`. A Googlebot group therefore replaces the `*` rules
   for Googlebot, it does not add to them.
2. Inside that group, the longest matching rule wins.
   `Allow: /shop/` beats `Disallow: /shop` for `/shop/shoes`.
3. When an Allow and a Disallow match with the same length, Allow wins.
4. `*` matches any run of characters, and `$` at the end anchors the
   rule to the end of the URL path.

Google Search Console's robots.txt report shows the last version Google
fetched and any parsing errors, which is the quickest way to see what
Google itself believes.

## Why isn't checking it once enough?

Because the file changes when something else changes: a deploy, a CMS
update, a new firewall rule. A check you run by hand describes the file
on the day you ran it. The changes that hurt are the ones nobody meant
to make, which means nobody thinks to check.

A cron job that fetches the file, diffs it against yesterday's copy,
and emails you on a difference covers the edits. It does not cover the
box running the cron going down, and a raw diff alerts on a comment
change as loudly as on `Disallow: /`.

## What should automatic robots.txt monitoring watch?

Four things, because they fail differently:

1. **Reachability.** A 5xx, a timeout, or a redirect loop on robots.txt
   pauses crawling. That is an outage, and it should page like one.
2. **Your critical paths.** Decide which paths must stay crawlable, at
   least `/`, maybe `/shop` or `/blog`, and alert the moment any of them
   is disallowed for the crawlers you care about.
3. **New Disallow rules**, with a diff, even when no critical path is
   blocked yet. A new rule is usually the first sign of a bad deploy.
4. **The file disappearing.** A missing robots.txt allows everything,
   so it is not an outage, but deliberate blocks and `Sitemap:` lines
   vanish with it.

A 401 or 403 on robots.txt deserves its own label. It is almost always
a firewall refusing the monitoring tool rather than a real rule, and
treating it as "the rules changed" sends you looking in the wrong place.

YoPingMe's [robots.txt monitor](/features/robots-txt-monitoring) covers
all four from your check regions and confirms a change on two checks in
a row before it alerts, so one odd response never pages anyone. It is
part of the Pro and Team plans. Its companion,
[sitemap monitoring](/features/sitemap-monitoring), watches the other
half of crawl health: how many URLs your sitemap lists.
