Measurement
Part of Log file analysis for technical SEO: what UK site owners should check
Log file analysis for technical SEO: what UK site owners should check
Log file analysis for technical SEO in the UK: what to check in server logs, from crawl frequency to bot user agents and UK GDPR retention rules.
What to take away
- Server logs record the requests that reached your origin, so they beat a crawl simulation when the two disagree.
- The first checks are status code mix, crawl frequency by directory, and whether real bot user agents appear.
- Log lines hold IP addresses, so UK GDPR appliespick a lawful basis and set a retention period.
- Many UK hosts keep raw logs for days rather than months, so export them before you start analysing.
Why server logs still matter
A crawler tool models how a site behaves. Server logs record what happened: the timestamp, the URL requested, the status code returned and the user agent behind each hit.
On cPanel and Plesk hosting, raw logs are usually a compressed file you download and parse locally. Managed WordPress platforms often summarise traffic in a dashboard and keep no raw lines, so ask your host what exists.
A five-step log analysis routine
Work through one week of logs in this order before changing anything on the site.
Five-step log analysis routine
- Verify bot identity against published IP ranges
- Count hits per verified bot per day
- Group requests by directory and template
- Tabulate status codes and note volume
- Compare log with Search Console crawl stats
- Verify bot identity. Match user agent strings against the published IP ranges for Googlebot and Bingbot, because scrapers copy those strings freely.
- Count hits per verified bot per day. A healthy site sees crawl frequency rise after new URLs appear in the XML sitemap.
- Group requests by directory and by template. Fixing a template that generates many URLs beats fixing pages one at a time.
- Tabulate status codes and note the volume in each band. That mix shows where crawl budget leaks.
- Compare the log with Google Search Console crawl stats, which report how often Google fetched each section.
For a longer view, exporting Search Console bulk data gives months of crawl history rather than single spikes.
What good crawl frequency looks like
Crawl frequency is a symptom, not a lever. It responds to publishing pace, internal linking, server speed and sitemap signals. A sudden drop usually points at a block, a slowdown or an accidental noindex.
Rising 5xx counts are the clearest warning. Search engines slow their crawl when a server errors under load, and recovery can take weeks after the errors stop.
Not every bot matters. Uptime monitors, price aggregators and SEO tools all leave traces, and some copy a Googlebot string. Filter for verified bots first.
Status codes worth counting
Sort requests by response code and note the volume in each band.
Status codes and actions
Code
- 200
- Page served
- 301 and 302
- Redirect
- 304
- Not modified
- 404
- Missing page
- 410
- Deliberately gone
- 429
- Rate limited
- 5xx
- Server error
Log meaning
- 200
- Confirm key URLs appear often
- 301 and 302
- Shorten chains, update internal links
- 304
- Normal for returning crawlers
- 404
- Fix link or accept removal
- 410
- Check URL meant to leave index
- 429
- Review host or CDN throttling
- 5xx
- Treat as incident, watch crawl rate
What to do
- 200
- 301 and 302
- 304
- 404
- 410
- 429
- 5xx
| Code | Log meaning | What to do |
|---|---|---|
| 200 | Page served | Confirm key URLs appear often enough |
| 301 and 302 | Redirect | Shorten chains and update internal links |
| 304 | Not modified | Normal for returning crawlers |
| 404 | Missing page | Fix the link or accept the removal |
| 410 | Deliberately gone | Check the URL was meant to leave the index |
| 429 | Rate limited | Review host or CDN throttling rules |
| 5xx | Server error | Treat as an incident, then watch crawl rate |
Keeping logs lawful under UK GDPR
An IP address is personal data, so a raw log file holds personal data records. You need a lawful basis to process it, and the retention and minimisation rules in the UK GDPR apply to log files as they do to customer records.
The ICO's UK GDPR guidance covers retention and security in practical terms. A workable policy for most sites: truncate or hash the final part of each IP address, keep full logs for around 30 days, keep aggregated counts for longer, and limit access to named staff.
Say so in your privacy notice. Retention periods should be stated, not implied.
Where UK hosts keep logs
Shared cPanel and Plesk plans normally expose raw access logs in combined format. VPS and dedicated servers running Nginx or Apache keep an access log that rotates on a schedule you control.
A CDN in front of the site changes the picture. If Cloudflare or a similar service terminates the connection, your origin sees the CDN rather than the visitor, and bot user agents may be rewritten or dropped. Ask whether the provider exports logs, and in what format.
Where the logs show crawl budget going to faceted or thin filtered URLs, crawl budget analysis with Screaming Frog sets out how to reconcile a log file with a controlled crawl.
Common questions
Does log file analysis need user consent?
No. Consent is one lawful basis among several, and log analysis usually rests on legitimate interests. You still owe visitors transparency, so describe log processing in your privacy notice.
How long should I keep raw log files?
For most UK sites, 30 days is a defensible default, with aggregated counts kept for up to a year. Whatever period you choose, you should be able to justify it.
Can a small site analyse logs without a developer?
Up to a point. A spreadsheet and a command-line filter handle a week of logs from a small site. Millions of lines need scripts or a paid tool.
What if my host keeps only three days of logs?
Set a recurring export rather than analysing ad hoc. A weekly download gives you a rolling record, and month-on-month comparison shows whether crawl behaviour is changing.
