Most enterprise catalogs are bigger than anyone thinks. Not the product count, which is known, but the URL count. Forty thousand SKUs across a few hundred categories, with filters for brand, size, material, price, rating, and availability, can produce millions of crawlable URLs. Almost none of them deserve to exist in a search index.
For years that was a crawl budget problem: Googlebot spent its time on filter combinations while new products waited to be discovered. In 2026 it is also an AI visibility problem. The crawlers feeding ChatGPT, Claude, and Perplexity behave differently from Googlebot, and faceted navigation built for one often fails the other. Here is where enterprise ecommerce sites go wrong, and how to fix each issue.
Why this got harder in 2026
Enterprise sites now serve two very different crawler populations.
Googlebot renders JavaScript. It fetches a page, queues it, runs the scripts, and indexes what a browser would see. Google's AI Overviews and AI Mode draw on that same infrastructure.
The major AI crawlers do not render. Large-scale analyses of AI crawler traffic, including one covering more than 500 million GPTBot fetches, found no evidence of JavaScript execution by GPTBot, ClaudeBot, or PerplexityBot. They read the HTML your server returns and leave. If your product grid, prices, or specs load client-side, those crawlers see an empty shell.
The AI companies also run several crawlers each, with different jobs. OpenAI operates GPTBot for model training, OAI-SearchBot for its search index, and ChatGPT-User for live fetches when someone asks about a page. Anthropic and Perplexity follow a similar pattern. Treating "AI bots" as one category in robots.txt or in your logs hides the decisions that actually matter.
Faceted navigation sits on top of both problems. It multiplies URLs for every crawler and, on many modern storefronts, it is the most JavaScript-dependent part of the site.
Issue 1: An infinite URL space nobody decided on
Google has been unusually direct about this. Its faceted navigation guidance names these URL spaces as the most common source of the crawling complaints it receives, and lays out two paths: if you do not need faceted URLs indexed, stop them from being crawled; if you do, make them behave.
The enterprise mistake is never choosing. Facets get added by merchandising, URL parameters get added by developers, and indexing behavior is whatever the platform does by default.
The fix: build a facet indexing matrix. List every facet and decide, deliberately, which combinations earn an indexable page. A useful rule: a facet combination deserves a page only when there is real search demand for it and enough products behind it to make a useful page. "Stainless steel ball valves" probably qualifies. "Stainless steel ball valves, four stars and up, in stock, sorted by price" does not.
- Index: category plus one high-demand attribute such as material, type, or brand, served as a clean static path with a unique title, intro copy, and a self-referencing canonical.
- Crawlable but not indexed: rarely, and only when you genuinely need link discovery through those pages.
- Not crawlable: sort orders, price sliders, ratings, availability toggles, multi-select combinations, and session or tracking parameters. Block them in robots.txt or move filter state into URL fragments so crawlers never request them.
For combinations you do index, Google's guidance adds practical hygiene: use the standard ampersand separator for parameters, keep filter order consistent so the same selection always produces the same URL, and return a 404 when a combination has no results instead of serving an empty page.
Canonical tags and nofollow on filter links help at the margins, but Google describes them as less effective over the long term than preventing the crawl. A canonical is a hint that still requires the crawler to fetch the page first.
Issue 2: Category and filter pages that render empty for AI crawlers
Many headless and modernized storefronts ship category pages as an application shell, then fetch products from a search API in the browser. Googlebot eventually sees the grid. GPTBot, ClaudeBot, and PerplexityBot see a heading and a loading state.
That matters because category and indexable facet pages are often exactly what AI tools want to cite for comparison questions like "best industrial label printers for warehouses." If the page has no products in its HTML, it has nothing to offer.
The fix: server-render what matters. For every indexable category and facet page, the initial HTML response should include product names, links, key attributes, prices or price ranges, intro copy, and structured data. Client-side JavaScript can still handle interactivity. The content just cannot depend on it.
Testing takes minutes. Compare view-source with the rendered page in browser DevTools. Anything visible only in DevTools is invisible to non-rendering crawlers. Then fetch a handful of key URLs with a plain HTTP client using an AI crawler user agent and read what comes back. The broader content tactics for AI citation are covered in our guide to LLM optimization.
Issue 3: A robots.txt written for 2019
Enterprise robots.txt files accumulate. A blanket block on AI bots gets added after a legal conversation, years-old facet rules apply only to Googlebot, and nobody owns the whole file.
The fix: make bot-by-bot decisions on purpose.
- Separate training from search. Blocking a training crawler like GPTBot is a reasonable business decision. Blocking search and user-triggered agents like OAI-SearchBot or ChatGPT-User removes you from the answers your buyers are reading. Decide each one with legal and marketing in the same room.
- Apply facet rules to every crawler. If filter parameters are disallowed for Googlebot but not for everyone else, AI crawlers spend their visits on the same useless combinations.
- Keep it owned. Assign an owner and review the file quarterly, because new crawler user agents keep appearing.
Issue 4: Conflicting canonical signals at scale
On large catalogs, canonical problems rarely come from one wrong tag. They come from systems disagreeing. The canonical on a filtered page points to the category. The XML sitemap lists the filtered URL. Mega menu links point to a third parameterized version. Regional storefronts each claim to be canonical.
Search engines resolve conflicts by guessing, and AI systems inherit whichever version wins, sometimes with a stale price attached.
The fix: one URL policy, enforced everywhere. Sitemaps should contain only indexable, canonical URLs that return a 200. Navigation and internal links should point to canonical URLs directly. Variant handling should be consistent: either one canonical product page with variant selection, or separate variant pages that each stand on their own, not a mix that changes by category. Multi-region sites need hreflang that matches the canonicals exactly.
Issue 5: Nobody looks at the logs
Most enterprise teams can tell you their rankings. Few can tell you what share of Googlebot's requests last month hit indexable URLs, or whether OAI-SearchBot ever fetched their top 500 product pages.
The fix: crawl analytics segmented by bot. Pull server or CDN logs monthly and report on a short list of measures:
- Share of crawl spent on indexable URLs versus parameters and facets
- Time from product launch to first crawl, by bot
- AI search and user-agent fetches on product and category pages versus everything else
- Response codes and response times served to crawlers, since slow or erroring pages get crawled less
If most of your crawl activity lands on filter URLs, you have found your first project.
A prioritized fix list
- Pull 30 days of logs and measure crawl distribution by bot and URL type.
- Build the facet indexing matrix with merchandising and SEO together.
- Block non-indexable facet parameters for all crawlers and enforce consistent filter ordering.
- Server-render product grids, specs, pricing, and structured data on indexable pages.
- Rewrite robots.txt with explicit decisions for training, search, and user-triggered agents.
- Align sitemaps, canonicals, internal links, and hreflang to one URL policy.
- Re-measure in 60 days and expand to the next set of facets.
FAQ
Q: Should enterprise ecommerce sites block AI crawlers? It depends on which ones. Training crawlers and search crawlers do different jobs. Many companies block training crawlers for content-rights reasons while allowing search and user-triggered agents, because those are what make products citable in AI answers. Blocking everything removes you from a research channel your buyers already use.
Q: Do AI crawlers render JavaScript? The major ones do not. Published analyses of GPTBot, ClaudeBot, and PerplexityBot traffic found no JavaScript execution. Google's AI features are the main exception because they rely on Google's own rendering infrastructure. Anything you want AI tools to read should be in the initial HTML.
Q: Is crawl budget only a concern for very large sites? Mostly, yes. A site with a few thousand URLs rarely needs to think about it. Enterprise catalogs with faceted navigation, multiple storefronts, and frequent product changes are exactly where crawl waste delays discovery of new and updated products.
The bottom line
Faceted navigation is a merchandising feature that quietly became an indexing policy. Decide which filter pages deserve to exist, make those pages readable without JavaScript, and give every crawler the same clear instructions. The result is a catalog that both Google and AI tools can understand, with crawl activity spent on the products you actually want found.
If you want a clear picture of how crawlers and AI tools see your catalog today, our SEO + AI Search Visibility Audit covers crawl distribution, rendering, and facet handling alongside how AI answers describe you. For the prompt-level view, see how to audit your presence in AI answers.