How to monitor sitemap changes
- seo
- sitemap
A sitemap tells search engines which URLs exist. When it breaks, it usually does not return an error. It returns a valid file with fewer URLs in it: a plugin update drops a post type, pagination stops after the first page, a sitemap index loses one of its child files. Nothing looks wrong unless you count. Monitoring a sitemap means counting its URLs on a schedule and noticing when the number moves.
How do I count the URLs in a sitemap?
For a single sitemap file, count the <loc> entries:
curl -sS https://example.com/sitemap.xml | grep -o '<loc>' | wc -l
If the URL ends in .gz, decompress it on the way:
curl -sS https://example.com/sitemap.xml.gz | gunzip | grep -o '<loc>' | wc -l
Check the status code too (curl -sS -o /dev/null -w '%{http_code}\n').
A sitemap answering 404 or 500 counts zero URLs, and so does a firewall
challenge page served with a 200.
What about a sitemap index?
Many sites, including most WordPress sites, publish a sitemap index: a
file of <sitemap><loc> entries, each pointing at a child sitemap. The
<loc> count of the index is the number of child files, not the number
of pages. To count pages, fetch each child and add up their <url>
entries:
curl -sS https://example.com/sitemap_index.xml \
| grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' \
| while read -r u; do curl -sS "$u" | grep -o '<url>' | wc -l; done \
| paste -sd+ - | bc
The sitemap protocol caps each file at 50,000 URLs and 50 MB uncompressed, which is why large sites split into an index in the first place.
How big a change is worth an alert?
Sitemaps change every time you publish, so "any change" is noise. Two rules keep the signal:
- Compare against a moving baseline, with a threshold. Pick a percentage, 20% is a reasonable start, and alert only when the count moves that far from the last count you accepted. Changes under the threshold become the new baseline, so steady publishing never adds up to an alert. A sudden swing does.
- Alert on growth as well as shrinkage. A sitemap that halves overnight lost pages. A sitemap that doubles overnight often means a crawl trap: faceted filters, session IDs, or duplicate URLs leaking into the file.
Confirm a change on a second fetch before alerting. A single truncated response or a cold cache can produce one odd count; two in a row is a real change.
What should automatic sitemap monitoring watch?
- That it loads: a 2xx, valid sitemap XML, and at least one URL. Anything else is an outage of the sitemap and should page like one.
- The URL count, following a sitemap index into its children, against a threshold and a moving baseline.
- From outside your infrastructure, so the check still runs when whatever broke the sitemap also broke the box that would have checked it.
YoPingMe's sitemap monitor does this
hourly on the Pro and Team plans. It follows a sitemap index up to 50
child files on the same host, reads .gz files, and alerts on a swing
past your threshold, 20% by default, in either direction. Pair it with
robots.txt monitoring, since a
sitemap full of URLs does not help if robots.txt blocks them.