Stopping bots crawling your filters
Filter pages multiply: every combination of category, brand and attribute is a URL, and the new generation of AI crawlers will walk every one of them. One customer’s server report showed a single bot-built URL combining thirteen brands taking over six seconds to serve – a page no human would ever request, that no cache had ever seen, being replayed against the server around the clock.
Super Speedy Filters defends against this in layers, and almost all of it is driven by your SEO settings – there is nothing extra to configure.
Available since 6.42.
What happens automatically
Deep pages stop doing the expensive work. Your SEO & Indexing settings (maximum indexable taxonomies, maximum indexable terms, and the rest) decide which filter combinations are worth indexing. Any page those rules mark as not-for-indexing defers all of its filter widgets until a real visitor interacts – the first mouse movement, scroll, tap or key press loads them. A crawler that renders the page but never touches it gets the products and pays almost nothing, because the filter panels never run.
Every visitor – human, Googlebot, scraper – receives identical HTML. The only difference is whether the client interacts. That matters: showing search engines different content from users is what gets sites penalised, and this design makes it impossible. Pages your rules mark as indexable are completely unaffected.
Dead filter URLs answer 410 Gone. Bots hold on to URLs they harvested long ago, including ones whose terms have since been renamed or deleted. A 404 tells a crawler “try again later” – forever – so these URLs answer 410 Gone instead, which polite crawlers treat as “delete this from your list”. Controlled by the Issue 404 for missing terms option on the Pretty URLs tab, on by default.
Filter links respect your indexing rules. Links that would lead beyond your indexable limits are marked nofollow and rendered as JavaScript-actioned links rather than plain crawlable ones, which keeps well-behaved crawlers inside the limits you set.
Two response headers your firewall can use
Every archive response carries two headers:
| Header | Example | Meaning |
|---|---|---|
X-SSF-Robots | noindex,nofollow | The page’s index/follow verdict under your SEO rules |
X-SSF-Filter-Depth | 13 | How many filter terms the request has selected in total |
Depth counts terms, not filters: thirteen brands in one brand filter is depth 13. Price, ordering and other non-taxonomy state are excluded.
Blocking the flood at the edge
A response header cannot stop the request that produced it – the first hit always reaches your server. The goal is to win the flood, not the first request, and there are three ways to use these signals, in order of how widely available they are.
1. Block deep URLs before they reach your server. For pretty filter URLs the depth is visible in the path itself, so a firewall rule can challenge them with zero server cost. A Cloudflare custom rule expression such as:
(http.request.uri.path matches "^/brand/([^/]+/){6,}")challenges any URL with six or more terms after /brand/, and the {6,} is the knob: {4,} challenges at four or more, {5,} at five. Six is deliberately cautious – a better starting point is your maximum indexable terms plus one, which for most stores means {4,}. Beyond your indexing limits a page is noindex anyway, so there is no search value to protect, and because the action is a challenge rather than a block, going lower costs almost nothing: the rare shopper who genuinely selects four terms answers one puzzle and carries on, while the bots stop three segments earlier. Adjust the /brand/ base to your own URL structure.
2. Count, then block. Rate limiting rules count matching responses per visitor and then block that visitor’s later requests. Counting on response status is widely available: a rule like “more than five responses with status 410 from one IP in a minute – block for an hour” cuts off any bot replaying a list of dead URLs after its first handful. On plans that support response-header matching, count on X-SSF-Robots containing noindex instead, which also catches deep-but-valid pages. A bot replaying ten thousand harvested URLs pays for five and gets refused the rest.
A ready-made Cloudflare rule for the depth header. Counting on response headers needs Cloudflare’s advanced rate limiting (an Enterprise feature – on other plans the 410 rule and the path rule above work the same way). Create a rate limiting rule with: If incoming requests match – your hostname, or narrow it to your shop paths; With the same characteristics – IP; When rate exceeds – 10 requests in 1 minute; Then take action – Managed Challenge for 1 hour. And this counting expression, which counts only responses whose filter depth is 5 or more:
any(http.response.headers["x-ssf-filter-depth"][*] matches "^([5-9]|[1-9][0-9]+)$")Header names are lowercase in Cloudflare expressions, and the regex reads as “5 to 9, or any number of two or more digits”. Tune the 5 to sit just above your indexing limits and what a real shopper plausibly selects. The action matters too: Managed Challenge means the rare human who genuinely multi-selects five terms answers one puzzle and carries on, while a bot replaying harvested deep URLs pays for its first ten and gets refused the rest.
3. No CDN? Use your server logs. The headers appear in your origin access logs (in nginx, $sent_http_x_ssf_robots and $sent_http_x_ssf_filter_depth), so tools like fail2ban – or your host’s own protection – can automatically ban IPs that accumulate noindex or 410 responses, with no CDN involved.
Do the firewall rule even if you do nothing else. Blocking obvious deep URLs before they reach WordPress costs nothing and removes the worst of the load immediately; everything else here tightens the net around what remains.
Watching it work: the Archive Performance report
The Archive performance report on the Performance tab shows who your archive pages are actually served to over the last fortnight – search engines, AI crawlers, SEO tools, other declared bots, and browsers – with request counts, total and average serving time, split by indexable versus noindex pages.
A large share of “browser” traffic hitting noindex pages is the signature of disguised scrapers: real browsers rarely reach deep filter combinations, and honest bots declare themselves. That row is your evidence for tightening the firewall rules above – and after you do, this report is where you watch the bot traffic decay.
Collection is deliberately lightweight: a handful of aggregate counters per day, recorded only on archive pages, self-pruning after thirty days. It cannot itself become a performance problem.
Comments