Crawl Budget for Small Sites: When It Matters and When It Doesn't

What crawl budget actually is

Crawl budget is the number of URLs Googlebot is willing and able to fetch from your site within a given period. It is set by two forces: crawl capacity, meaning how many requests your server can handle without slowing down, and crawl demand, meaning how much Google wants your content based on its popularity and how often it changes. Neither is a figure you can look up somewhere; Google adjusts both continuously for every site it knows about.

The term escaped from enterprise SEO, where teams managing millions of URLs genuinely have to ration Googlebot's attention, and it trickled down to small-business owners who mostly do not need it. Google's own documentation is unusually direct about who should care: sites with over a million pages, or sites with tens of thousands of pages whose content changes very rapidly. It also states plainly that sites with fewer than a few thousand URLs are generally crawled efficiently with no intervention at all.

The maths for a typical small business website

Take an honest inventory of a typical UK small-business site and the numbers are tiny. A trades business or salon site usually runs to 15 to 60 pages. A professional services firm with a busy blog might reach 150. Even a modest online shop with 300 products, 30 categories and 50 articles sits comfortably under 500 indexable URLs.

Googlebot routinely crawls sites of that size in full, often revisiting changed pages within days. If your new blog post takes a fortnight to appear in Google, the cause is almost never crawl budget. It is far more likely to be a quality threshold, a weak internal linking structure, or a page Google has fetched and simply chosen not to index yet, which is a different problem with different fixes.

When a small site genuinely has a crawl problem

The exception matters: crawl budget is about the URLs your platform generates, not the pages you deliberately publish. Certain features can multiply a 300-page site into hundreds of thousands of crawlable addresses without anyone noticing.

• Faceted navigation, where every combination of size, colour, brand and price filter creates a new URL such as ?colour=red&size=10&sort=price
• Indexable internal search results, where every query a visitor types becomes a crawlable page
• Calendar or booking widgets with 'next month' links stretching endlessly into the future
• Session IDs and tracking parameters appended to internal links by plugins or analytics tools
• Printer-friendly, currency or sort-order duplicates of every product page

This is why the problem hits small ecommerce sites hardest. WooCommerce, Magento and some Shopify themes generate filter URLs freely, and Googlebot can spend an entire visit fetching near-identical variants while your genuinely new pages wait their turn.

Diagnosing it in Google Search Console

Search Console answers the question in about ten minutes, and two reports do most of the work.

• Open Settings, then Crawl stats. Check total crawl requests, average response time and the breakdown by file type; a healthy small site shows most requests hitting HTML pages you recognise
• Open Indexing, then Pages, and study the not-indexed reasons: large counts under 'Alternate page with proper canonical tag' or 'Duplicate without user-selected canonical' usually point to parameter URLs
• Watch 'Discovered - currently not indexed': if it holds thousands of URLs on a small site, Google is finding far more addresses than your content justifies
• Sample the listed URLs; if they are filter combinations or search results rather than real pages, you have found the leak

The URL Inspection tool at the top of Search Console is useful for spot checks: paste in a suspicious address and it tells you when Google last crawled it and which page it treats as the canonical. For a second opinion, crawl your own site with Screaming Frog's free tier and compare its URL count with the number of pages you believe you have. A big gap between the two figures is the clearest signal you will get.

Fixes that actually work

If the diagnostics show a URL explosion, the remedies are well established and mostly quick.

• Block wasteful facet parameters in robots.txt, for example Disallow: /*?colour=, so Googlebot stops fetching the combinations
• Add noindex to internal search results pages, and let your platform's canonical tags point filter variants at the parent category
• Trim your XML sitemap to canonical, 200-status URLs only, so it reads as a clean statement of what matters
• Collapse redirect chains and fix soft-404s, both of which waste fetches
• Keep server response times sharp, because slow responses cause Google to throttle its crawl rate

One caution: robots.txt stops crawling, not indexing. A blocked URL with links pointing at it can still appear in results as a bare address. Use robots.txt for wasteful parameter spaces and noindex for pages Google should fetch but never list, and avoid applying both to the same URL, because a noindex tag on a blocked page can never be read.

Where your energy is better spent

For the overwhelming majority of UK small businesses, crawl budget belongs at the bottom of the SEO list. Google can fetch your site without effort; the real contest is whether your pages deserve to rank once fetched. Better uses of the same afternoon include tightening title tags, adding internal links from your strongest pages to your money pages, thickening thin service pages and gathering reviews for local visibility.

Treat crawl diagnostics as an annual health check rather than a weekly obsession, and escalate only if your platform starts spawning URLs you never created. If you would rather have a specialist read the reports, our team at Thind Global Services can run that check as part of a wider technical audit.