How to Use Screaming Frog for Crawl Budget Analysis on UK Websites

What to take away

Screaming Frog cannot see Googlebot's crawl, but it shows which URLs consume capacity without earning traffic.
Match the crawler's user agent and rendering mode to Googlebot before reading any report.
Judge waste by directory, parameter and status code, not by site-wide totals.
Confirm the crawl against server logs, because the two fetch different sets of URLs.
Clear waste on existing URLs before publishing more pages.

What crawl budget actually means

Crawl budget is the number of requests Googlebot can make to your host in a set period. Google splits it into a crawl rate limit, which depends on server response speed, and crawl demand, which depends on how much Google wants your URLs. You cannot raise the limit by asking. You can only stop spending it on URLs that will never rank.

Crawl budget components

Crawl rate limit

Depends on: Server response speed
Can you raise it?: No, only improve speed
UK impact: London hosting adds latency

Crawl demand

Depends on: Google's desire to crawl
Can you raise it?: No, only earn demand
UK impact: Large sites hit ceiling first

UK sites feel this unevenly. Hosting in one London data centre adds latency for Googlebot crawling from outside the UK, and slow responses reduce the rate limit Google applies. A retailer with thousands of product variations, or a publisher with tag archives, hits that ceiling long before a brochure site does.

Configure the crawl to mirror Googlebot

Start from the domain root, not an XML sitemap. Internal linking, orphan pages and redirect chains only appear when the crawler follows links.
Set the user agent to Googlebot Smartphone under Configuration, then User Agent. Screaming Frog sends that string from your own IP address, so the result is a simulation.
Enable JavaScript rendering under Config, Spider, Rendering. Use the rendered crawl only when templates need it, because it inflates the request count.
Limit crawl speed to roughly the average response time Search Console reports for your property.
Keep respect for noindex directives switched on, so the crawl reflects what Google can index.
Exclude analytics, tag manager and CDN hosts, otherwise third-party requests dominate your crawl statistics.

Read crawl waste by URL pattern

Sort the internal HTML by crawl depth and status code. Depth five or more on a retail site usually means filter combinations. Check the Near Duplicates report, then the Directives tab for canonicals pointing elsewhere.

Crawl waste patterns and fixes

Pattern

Facet links: ?colour=, ?size=, ?sort=
Tracking params: ?utm_, ?sid=, affiliate IDs
Deep pagination: /page/12/ and deeper
Redirect chains: 301 pointing to another 301
Soft 404s: 200 responses for withdrawn products
Crawlable endpoints: /api/, /json/, /cart

Meaning

Facet links: Canonical to parent, block robots.txt
Tracking params: Strip at server, link clean URL
Deep pagination: Self-referencing canonicals, leave indexable
Redirect chains: Update internal links to final URL
Soft 404s: Return 404 or 410 once stock gone
Crawlable endpoints: Disallow in robots.txt

Action

Facet links
Tracking params
Deep pagination
Redirect chains
Soft 404s
Crawlable endpoints

Pattern in the crawl	What it usually means	Action
?colour=, ?size=, ?sort=	Facet links followed from every product	Canonical to the parent category, then block in robots.txt
?utm_, ?sid=, affiliate IDs	Tracking parameters in internal or feed links	Strip at the server so the clean URL is linked
/page/12/ and deeper	Pagination beyond the point anyone browses	Keep self-referencing canonicals, leave indexable
301 pointing to another 301	Legacy migration left in place	Update internal links to the final URL
200 responses for withdrawn products	Soft 404s	Return 404 or 410 once stock is gone
/api/, /json/, /cart	Crawlable endpoints that need no index	Disallow in robots.txt

Compare those patterns per directory rather than against the site-wide figure. An average response time of 400 milliseconds hides a category directory answering in two seconds, and per-directory speed shapes the crawl rate Google applies.

For a checklist of the wider ecommerce problems these crawls expose, see the technical SEO audit checks UK ecommerce sites often miss.

Confirm the crawl against server logs

Screaming Frog shows what could be crawled. Server logs show what Googlebot did crawl. Export Googlebot requests, group them by directory, and compare totals with the crawl you ran. Heavy log traffic and no organic sessions in the same directory is the clearest sign of waste.

The Crawl Stats report in Google Search Console gives Google's own view of requests, response times and status codes for your host.

Log files hold IP addresses and full request strings, which count as personal data under the UK GDPR. The ICO's guide to the UK GDPR sets out the lawful basis and retention limits to settle before you keep extracts.

Working through hits line by line is faster with a repeatable process, and our guide to log file analysis for technical SEO covers the filters that matter.

Cut waste before you add pages

Where two URLs serve the same content, pick one and consolidate with a canonical tag. Google's guidance on consolidating duplicate URLs explains how signals are merged.

Fix the internal links pointing at the duplicate, not just the tag. Canonical is a hint, and Google can ignore it when internal links and sitemaps disagree. Blocking a parameter in robots.txt also prevents Google from reading the canonical on that URL.

Common questions

Does Screaming Frog measure crawl budget?

No. It measures site structure. Use it to find URLs that waste crawl capacity, then check the logs to see whether Googlebot's requests fall.

How do I know which URLs Googlebot crawls?

Server logs are the only direct evidence. Verify the crawler by reverse DNS, then group requests by directory and status code.

Should I block faceted URLs in robots.txt?

Only patterns you are certain should never be indexed. If a facet earns traffic, canonicalise it instead, because a robots.txt block hides the canonical hint from Google.

How often should I rerun the crawl?

Monthly for most sites, and after any template, migration or pagination change. Compare crawl depth and status codes between runs to see whether waste is falling.