Technical SEO vendor due diligence, a checklist or a leap of faith?

What to take away

A Manchester agency choosing a crawler for a 40,000-page retail site needs evidence, not a demo: ask for a sample crawl export and a written method.
Check status code handling against Google's documented behaviour before you accept any index diagnostics.
Treat procurement risk as real, because technology buying fails on process as often as on features.
Agree KPIS before the pilot, using a written method for defining and reporting success.
Keep the exit clause on the checklist, because migration cost is where English teams lose budget.

Crawl and index evidence

Start with what the tool actually fetches. A vendor claiming full JavaScript rendering should show you a crawl of a client-like site, with a log of blocked requests.

Crawl evidence to request

Client-like crawl with blocked request log
Explanation of soft 404 handling
Raw exportURLs, status codes, timestamps
Crawl budget per host
Robots.txt and CDN rate limit respect

Ask how the crawler treats 4xx and 5xx responses. Google's guidance on how HTTP status codes affect its crawlers is the reference point: a vendor that cannot explain soft 404s will misreport your index coverage.

Request one raw export, not a dashboard screenshot. You want URLS, status codes, timestamps and crawl budget per host. If the export is unavailable, that is your answer.

Ask whether the crawl respects your robots.txt rules and your CDN rate limits. A tool that ignores them sends you chasing problems you do not have.

Procurement and contract checks

Pricing models hide risk. Seat-based licences punish growth. Crawl-credit models punish large sites. Both need a ceiling written into the contract.

The US Government Accountability Office's report on cloud computing procurement challenges (GAO-26-107530) covers requirements, competition and contract terms. Apply the same scepticism to, for example, a £900 a month crawler subscription.

Term and renewal notice period.
Data ownership and export format.
Subprocessor list and change notification.
Service credits for downtime.
Exit assistance, priced or included.

Data protection questions

If the crawler sends URLS to a third-party AI service, that is a data flow you must document. Ask for the subprocessor list and the retention period.

Under UK GDPR, your controller obligations do not transfer to the vendor. A processor agreement is the minimum, not the whole answer.

For a wider view of how supplier tools fit together, read the technical SEO tools and supplier guide before the security review meeting.

Pilot scoring before signature

Run a two-week pilot on a staging copy of your site. Score it against fixed criteria, not vendor slides.

The GOV.UK service manual explains how to measure the success of your service. Use that method to define four KPIS: crawl coverage, false positive rate on broken links, time to first useful report, and support response time.

A workable example: a Leeds team paying £600 a month sets a target of 95 per cent URL coverage on a 20,000-page site. If the pilot returns 70 per cent, the price is irrelevant.

Reference call questions

Speak to two existing customers of similar site size. Ask what broke, not what worked.

How long did onboarding take against the plan?
What did the first invoice include that the quote did not?
Who fixed the crawl configuration error?
What would you negotiate differently?

Record the answers. They belong in the same file as the technical SEO software selection notes, so the decision survives staff changes.

Glossary

Crawl budget: requests a search engine allocates to your host.
Soft 404: a page returning 200 but behaving like a missing page.
Processor agreement: contract term governing personal data handled for you.
Subprocessor: a third party the vendor uses to deliver the service.
Exit assistance: migration support written into the contract.

Common questions

How long should due diligence take?

Two to four weeks for a mid-market site, including a pilot and two reference calls. Compressing it to days usually shifts the cost to onboarding.

Do we need a pilot if the vendor is well known?

Yes. Brand recognition says nothing about how the crawler handles your JavaScript framework or your CDN rules.

What is the strongest signal?

A willingness to share a raw crawl export and a named subprocessor list before signature. Refusal on either point is a stop signal.

Can we reuse this checklist for a consultancy?

Mostly. Replace crawl coverage with deliverable acceptance criteria, and keep the exit terms unchanged.