Yo! your robots.txt just blocked Google.
One line in robots.txt can take a whole site out of search, and the homepage stays a perfectly healthy 200 while it happens. A robots.txt monitor reads the file the way a crawler does and tells you the moment it starts saying no.
- Unreachable or 5xx
- A critical path blocked
- New Disallow rules
- robots.txt removed
Why watch robots.txt at all?
Because the failures are silent. A staging robots.txt with Disallow: / gets deployed to production. A CMS plugin update rewrites the file. A firewall starts answering crawlers with 403. Nothing breaks for visitors, so nothing alerts, and you find out weeks later from a traffic graph. A server error is its own problem: Google's documentation says that while robots.txt answers with a 5xx, Googlebot stops crawling the site for the first 12 hours, then falls back to the last copy it cached. One small file failing is enough to pause your whole site's crawl.
What sends an alert
Two kinds of alert, sent to the same email, webhook, and Slack channels as everything else.
DOWN, the same as any outage, confirmed from a second region before anyone is paged:
- robots.txt cannot be reached, times out, or answers with a 5xx or 429.
- A critical path is disallowed for a crawler you watch. By default that means the site root,
/, for*, Googlebot, and Bingbot. Add your own paths, like/shopor/blog, up to 20, and up to 10 crawlers. - robots.txt answers 401 or 403. That is almost always a firewall turning our probe away rather than a real rule, and allowlisting the probefixes it. It is reported on its own so it never reads as "your rules changed".
The monitor stays down until the file is fixed, then you get a RECOVERED.
CHANGED, for edits that matter even when nothing is fully blocked yet:
- New Disallow rules for a crawler you watch. The alert lists the added rules and includes a line diff of the old and new file.
- robots.txt was removed or emptied. Crawlers read a missing robots.txt as "everything allowed", so this is not an outage, but any
Sitemap:lines and deliberate blocks just went with it. - Optionally, any edit at all. Tick "also alert on any edit" and every change arrives with its diff. Leave it off and those edits are still recorded, quietly.
How do you avoid false alarms?
A change has to be seen on two checks in a row before it counts, from any of your check regions. One odd response, or a CDN that serves two versions of the file from different edges, never sends a CHANGED alert. The first check only records the current file, so adding a monitor never alerts about rules that were already there.
How the rules are read
Matching follows RFC 9309, the robots.txt standard Google and Bing implement: a crawler uses the group named after it, or the * group when none is; the longest matching rule wins; Allow wins a tie; * and a trailing $ work as wildcards. The probe reads up to 512 KiB of the file, a little past the 500 KiB Google itself reads, so rules Google ignores cannot raise an alarm.
Setting one up
In the dashboard, add a monitor and choose robots.txt. Paste any URL on the site; the monitor always checks /robots.txt at the site's root, because that is the only place crawlers look. It checks every 15 minutes by default, and as often as every 5. The monitor page keeps the last 100 changes, each with its diff, so "who blocked /checkout and when" has an answer.
Which plans include it
robots.txt monitors are part of the Pro and Team plans, and each one counts as a monitor in the plan's allowance. You can upgrade from your account page; both plans start with a 14-day trial, one per account. The free plan keeps its HTTP, keyword, SSL, ping, and TCP checks.
Watching how many pages your sitemap lists is the companion check:sitemap monitoring. For doing it by hand, the guide how to monitor robots.txt changes has the commands.
Start at app.yoping.me, then add the robots.txt monitor once you are on Pro or Team.