Large Sites and Performance
Super Speedy Sitemaps is built for sites with hundreds of thousands or millions of URLs. A sitemap plugin that builds one huge file per request runs out of memory or time on a catalogue that size, and the visitor (or Googlebot) gets a white page. This plugin avoids that with four mechanisms: small files, batched database queries, fast pagination and a disk cache.
Item limits
XML Sitemaps > Item limits > Global limit caps how many URLs go in one sitemap file. The default is 5,000.
The sitemap protocol allows up to 50,000 URLs per file, but a higher limit means more work in a single request. On a large catalogue that is what produces white pages, 504 Gateway Timeout errors and “Allowed memory size exhausted” errors. Search engines handle many small files just as well as a few big ones, so there is nothing to gain from raising the limit.
Keep the Global limit at 5,000 or lower. If a sitemap times out on your server, lower it to 2,000 before trying anything else.
Splitting post-based sitemaps
Split post-based sitemaps is on by default. When a post type has more URLs than the limit, its sitemap is split into numbered parts: post_product_part1.xml, post_product_part2.xml and so on. Every part is listed in the sitemap index, so you still submit only one URL.
Split limit sets a separate per-file limit for the split parts. Leave it at 0 to use the Global limit.
Each part only fetches its own slice of posts. The query for part 40 resolves the matching post IDs first and fetches the full rows only for those IDs (a deferred join), so deep parts avoid the slowdown a plain OFFSET query suffers on a large posts table.
Database query limit
Advanced Options > Sitemap modules > Database query limit sets how many rows each database query fetches while a sitemap is built. The default is 1,000. A 5,000-URL sitemap is therefore built from five small queries rather than one large one, which keeps peak memory flat however big the site is.
Lower it (for example to 500) only if a host with very little memory still fails at a Global limit of 2,000.
Sitemap cache
Advanced Options > Sitemap Cache controls the disk cache:
| Setting | Default | What it does |
|---|---|---|
| Enable caching | Off | Save each generated sitemap to disk and serve it from there |
| Enable auto cache re-generation | On | Rebuild an expired file on the next request |
| Cache expiry time | 1 hour | How long a cached file is served before it is rebuilt |
| Cache directory | wp-content/uploads/super-speedy-sitemaps | Where the files are saved (network super admin only on multisite) |
Turn caching on once your sitemap contents are settled. Without it, every request for every sitemap runs the database queries again, which on a large site means a crawler fetching 200 sitemap parts triggers 200 builds.
Click Flush Cache to delete the cached files. The next request for each sitemap rebuilds it. The cache is also flushed automatically when The SEO Framework or Super Speedy Filters changes which URLs are indexable.
A page cache or CDN in front of your site (Cloudflare, Varnish, a host’s edge cache) can keep serving an old copy of a sitemap after you flush the plugin’s cache. Purge the sitemap URLs there too, or exclude *.xml sitemap URLs from the page cache.
Compression
Advanced Options > Formatting > Compress sitemaps gzips each sitemap, making it around 70% smaller. Most web servers already compress XML responses, and compressing twice breaks the response with a “Content Encoding Error”. Leave it off unless you know your server does not compress XML. See Troubleshooting.
Image sitemaps cost extra queries
The Google Image Sitemap extension adds each post’s featured image to its sitemap entry. That needs extra lookups per URL. If a large sitemap becomes slow after enabling it, that extension is the first thing to switch off. See Google News and Image Sitemaps.
Measuring a build
With Advanced Options > Debugging > Enable build stats on (the default), the last line of each sitemap is a comment such as This sitemap was originally generated in 0.412 second(s) (Memory usage: 18.5 MB) - 7 queries - 5000 URL(s) listed. View the sitemap’s page source to see it, and compare it before and after a settings change. On a cached sitemap the figures are from the build that created the cached file.
Sitemap Log lists every sitemap generated, when, and any errors.
Further reading
Written for Super Speedy Sitemaps 1.11.17.
Comments