Every unique combination of WooCommerce filter parameters — ?color=blue&size=l, ?color=blue&size=m, and so on — is, to a crawler, a brand new URL. Google’s own crawling documentation calls this “faceted navigation” and warns it can create an “infinite URL space” that wastes crawl budget on near-duplicate pages. InstantFilter’s filter click generates no new URL at all, because the click never leaves the browser.
What does Google mean by an “infinite URL space”?
Google’s crawling infrastructure documentation describes the mechanism directly: “Changing any of the URL parameters… would show a different set of items on the underlying page. This often means a very large number of possible combinations of filters, which translates to a very large number of possible URLs.” Three color options times four sizes times six brands is 72 URLs for one category — and that number multiplies across every category in the catalog.
The cost Google names has two parts, in its own words: “Overcrawling: Because the URLs created for the faceted navigation seem to be novel and crawlers can’t determine whether the URLs are going to be useful without crawling first, the crawlers will typically access a very large number of faceted navigation URLs before the crawlers’ processes determine the URLs are in fact useless,” and “Slower discovery crawls: Stemming from the previous point, if crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs.” If you are searching woocommerce filter urls duplicate content or crawl budget faceted navigation, this is the exact mechanism: not a penalty, a budget problem.
How do you block crawling of filtered URLs with robots.txt?
If you don’t need filtered combinations to rank on their own — the common case for a WooCommerce catalog, where the unfiltered category page and individual product pages carry the SEO value — Google’s documented fix is to block the parameters directly in robots.txt:
user-agent: Googlebotdisallow: /*?*color=disallow: /*?*size=allow: /*?products=all$
Google frames the trade-off plainly: “Oftentimes there’s no good reason to allow crawling of filtered items, as it consumes server resources for no or negligible benefit; instead, allow crawling of just the individual items’ pages along with a dedicated listing page that shows all products without filters applied.” That dedicated listing page — the plain category URL with no query string — is what you keep crawlable; every filtered permutation of it is what you block.
Why does Google call rel=canonical “less effective in the long term”?
Canonical tags are the alternative when you can’t or don’t want to touch robots.txt: point every filtered URL’s rel="canonical" at the unfiltered category. Google’s own documentation is explicit about the ceiling on this approach: “Using rel=\”canonical\” to specify which URL is the canonical version of a faceted navigation URL may, over time, decrease the crawl volume of non-canonical versions of those URLs,” immediately followed by the caveat that this and rel="nofollow" “are generally less effective in the long term than the previously mentioned methods” — meaning less effective than blocking crawling outright in robots.txt. A canonical tag is a hint Google weighs, not an instruction it is bound to follow; a disallow rule in robots.txt stops the crawl before it starts.
rel="nofollow" on filter links has its own sharp edge, named directly in the same documentation: “every anchor pointing to a specific URL must have the rel=\”nofollow\” attribute in order for it to be effective.” One filter link anywhere on the site without that attribute — in a sitemap, a related-products widget, an old blog post — reopens the path Google can crawl.
What about URL fragments instead of query parameters?
Google names a third option, separate from blocking or canonicalizing: move the filter state into a URL fragment instead of a query parameter — #color=blue&size=l instead of ?color=blue&size=l. The documentation states the effect directly: “Google Search generally doesn’t support URL fragments in crawling and indexing. If your filtering mechanism is based on URL fragments, it will have no impact on crawling (positive or negative).” That is a structural way to opt out of the faceted-navigation problem for a plugin architecture that still wants to reflect filter state in a shareable link, without touching robots.txt or canonical tags at all — though it also means a fragment-based filtered URL cannot be targeted for ranking either, since Google doesn’t crawl or index what follows the #.
If you DO want filtered pages indexed, what does Google require?
Some stores have a real reason to rank filtered pages — “men’s running shoes in size 10,” say, if that combination gets meaningful search volume on its own. Google’s documentation sets three concrete requirements for that path. First, use the standard parameter separator: “Use the industry standard URL parameter separator ‘&’. Characters like comma, semicolon, and brackets are hard for crawlers to detect as parameter separators.” Second, if filters live in the URL path rather than the query string — /products/fish/green/tiny instead of ?color=green&size=tiny — “ensure that the logical order of the filters always stays the same and that no duplicate filters can exist,” so the same combination never produces two different-looking URLs.
Third, and most often skipped: “Return an HTTP 404 status code when a filter combination doesn’t return results… don’t redirect to a common ‘not found’ error page. Instead, serve a ‘not found’ error with the 404 HTTP status code under the URL where it was encountered.” A filter combination with zero matching products that quietly returns a 200 with an empty grid, or a soft redirect to the homepage, is exactly the kind of “useless” URL Google’s overcrawling warning describes — the crawler has no signal to stop requesting it.
Why doesn’t WooCommerce’s native filter architecture decide this for you?
Because the URL behavior is a side effect of whichever filter architecture runs the click, not a setting WooCommerce core exposes. A classic AJAX filter plugin appends a query string on every click — the architecture covered on the admin-ajax.php page in this cluster — and that query string is exactly the URL space Google’s faceted-navigation guidance addresses. An index-based AJAX plugin does the same thing faster, but the URL still changes per combination. Whether those URLs get crawled, blocked, or canonicalized is then a separate decision every store has to make on top of whichever filter plugin it runs; it is not automatic.
Why doesn’t InstantFilter generate this problem at all?
Because the filter click never produces a server request or a new URL to begin with. After the archive hydrates, InstantFilter filters the already-downloaded codebook in the browser; the category’s own guide already states this directly: “Omdat de filters geen duizenden unieke AJAX-URL’s of query-strings genereren die de bot moet proberen te volgen, blijft de indexering van je shop schoon en gefocust op je belangrijkste pagina’s” — because the filters don’t generate thousands of unique AJAX URLs or query strings for the bot to try to follow, the site’s indexing stays clean and focused on its main pages. There is no faceted-navigation decision to make, because there is no faceted URL space in the first place: one category has one crawlable URL, filtered or not, and Googlebot sees the same fully rendered HTML grid either way thanks to server-side rendering on the first load.
Keep going
The URL a filter click produces is an architecture decision. These pages cover the rest of it.
- AJAX versus frontend-first filtering architecture
- The request behind a query-string filter click
- Why query strings bypass your page cache too
- Why WooCommerce filters get slow
- Pricing and trial
