Yo! your robots.txt just blocked Google.

One line in robots.txt can take a whole site out of search, and the homepage stays a perfectly healthy 200 while it happens. A robots.txt monitor reads the file the way a crawler does and tells you the moment it starts saying no.

Why watch robots.txt at all?

Because the failures are silent. A staging robots.txt with Disallow: / gets deployed to production. A CMS plugin update rewrites the file. A firewall starts answering crawlers with 403. Nothing breaks for visitors, so nothing alerts, and you find out weeks later from a traffic graph. A server error is its own problem: Google's documentation says that while robots.txt answers with a 5xx, Googlebot stops crawling the site for the first 12 hours, then falls back to the last copy it cached. One small file failing is enough to pause your whole site's crawl.

What sends an alert

Two kinds of alert, sent to the same email, webhook, and Slack channels as everything else.

DOWN, the same as any outage, confirmed from a second region before anyone is paged:

The monitor stays down until the file is fixed, then you get a RECOVERED.

CHANGED, for edits that matter even when nothing is fully blocked yet:

How do you avoid false alarms?

A change has to be seen on two checks in a row before it counts, from any of your check regions. One odd response, or a CDN that serves two versions of the file from different edges, never sends a CHANGED alert. The first check only records the current file, so adding a monitor never alerts about rules that were already there.

How the rules are read

Matching follows RFC 9309, the robots.txt standard Google and Bing implement: a crawler uses the group named after it, or the * group when none is; the longest matching rule wins; Allow wins a tie; * and a trailing $ work as wildcards. The probe reads up to 512 KiB of the file, a little past the 500 KiB Google itself reads, so rules Google ignores cannot raise an alarm.

Setting one up

In the dashboard, add a monitor and choose robots.txt. Paste any URL on the site; the monitor always checks /robots.txt at the site's root, because that is the only place crawlers look. It checks every 15 minutes by default, and as often as every 5. The monitor page keeps the last 100 changes, each with its diff, so "who blocked /checkout and when" has an answer.

Which plans include it

robots.txt monitors are part of the Pro and Team plans, and each one counts as a monitor in the plan's allowance. You can upgrade from your account page; both plans start with a 14-day trial, one per account. The free plan keeps its HTTP, keyword, SSL, ping, and TCP checks.

Watching how many pages your sitemap lists is the companion check:sitemap monitoring. For doing it by hand, the guide how to monitor robots.txt changes has the commands.

Start at app.yoping.me, then add the robots.txt monitor once you are on Pro or Team.