How Super Speedy Sitemaps Scales
Most sitemap plugins are written for blogs. On a WooCommerce store or a content site with hundreds of thousands of posts, building a sitemap means reading a large part of wp_posts, and the way a plugin asks for those rows decides whether the sitemap loads in a second or times out. This article explains what Super Speedy Sitemaps does differently, so you can see why it stays fast as a site grows and which settings matter.
Where the plugin comes from
Super Speedy Sitemaps started as a fork of Better WordPress Google XML Sitemaps. In 2019 Dave Hilditch compared Yoast SEO’s sitemaps, Google XML Sitemaps and Better WordPress Google XML Sitemaps on a very large client site, as part of a performance analysis. Better WordPress Google XML Sitemaps was the only one of the three whose design could be made fast at that size: it splits large sitemaps into pages and lists them in an index, where the others either built single huge files or ran queries no database index can help. The write-up, with the queries and the indexes that fixed them, is Performance Optimisation for various XML Sitemap plugins.
Better WordPress Google XML Sitemaps was no longer maintained, so it became Super Speedy Sitemaps: the security problems fixed, the deferred-join query from that article built in, and the plugin maintained and tested against current WordPress.
One index, many small files
Search engines are given one address, /sitemap.xml, the sitemap index. The index holds no page URLs itself. It lists the child sitemaps:
site.xmlfor the home page- one sitemap per public post type, such as
post.xml,page.xmlandpost_product.xml - one sitemap per enabled taxonomy, such as
taxonomy_product_cat.xml - Google News, image, external page, date and author sitemaps when you switch them on
A post type with more URLs than the split limit is published as numbered parts instead: post_product_part1.xml, post_product_part2.xml and so on. The index works out how many parts each post type needs from one count query grouped by post type, so building the index never reads the posts themselves.
Each file holds 5,000 URLs by default. The sitemap protocol allows 50,000, but a file that size takes far longer to build and needs far more memory, and it is what produces white pages and timeouts on large catalogues. Smaller files cost nothing: search engines read the index and fetch every part.
Fetching the rows
A child sitemap reads its posts in batches, 1,000 rows per query by default (Advanced Options > Sitemap modules > Database query limit), so memory use stays flat however large the post type is.
Each batch uses a deferred join. A plain query for, say, part 120 of a product sitemap asks the database for rows 595,000 to 600,000 ordered by last-modified date, and the database reads every row before them to find the page. Super Speedy Sitemaps asks for just the post IDs of that page first, which the database can answer from an index, then fetches the full rows for those IDs only, so reaching a deep part does not mean reading every row before it.
Scalability Pro adds a wp_posts index built for exactly this query: post type, status, password and last-modified date. Super Speedy Sitemaps works without it, but on a large catalogue the index lets every batch be read straight from the index rather than by scanning the table.
Building once, serving many times
With Advanced Options > Sitemap Cache > Enable caching on, each sitemap is built once, saved gzipped to wp-content/uploads/super-speedy-sitemaps/ and served from disk until it expires (one hour by default). A crawler that reads 40 sitemap files triggers at most 40 builds per expiry period, not one per request. Cached files carry Last-Modified and ETag headers, so a crawler that already has a file gets a 304 Not Modified answer with no body at all.
With Enable auto cache re-generation on, an expired file is rebuilt on the next request for it. To build every file before a crawler asks, run wp sssm warm from the server’s cron.
When the server is too busy
A busy server can fail part-way through building a sitemap: the database drops a query, or PHP runs out of memory or time. Super Speedy Sitemaps checks every build for database errors and PHP fatal errors, and a build that hit one is never cached or served. Instead the visitor gets one of:
- the last good copy, the cached file from the last successful build, kept on disk even after it expires for exactly this purpose, or
503 Service Unavailablewith aRetry-Afterheader, when there is no copy. Search engines treat a 503 as “come back later” and keep the URLs they already have.
Every failure is recorded in the Sitemap Log with the error. Advanced Options > Sitemap Cache > When a sitemap cannot be generated chooses between serving the last good copy (the default) and always responding 503.
Turn on caching on any live site. It is what keeps crawlers from rebuilding sitemaps on every request, and it is what provides the last good copy when the server is under load. Caching is off by default only so that changes show up instantly while you set the plugin up.
What it writes to your database
Sitemaps are files, not database rows. The plugin stores its settings as WordPress options and keeps one table of its own, the generation log behind the Sitemap Log, capped at the last 1,000 builds. Serving a cached sitemap writes nothing to the database.
Further reading
Written for Super Speedy Sitemaps 1.16.0.
Comments