When a Site Blocks the Audit

Some sites refuse to serve anything that is not a person in a browser. When that happens, SitePulse stops before scoring rather than reporting a page it never read. The audit is marked Failed, and a badge says why.

Two different failures

Bot protection shows as Bot protection · Vercel (or Cloudflare, or DataDome). The site's firewall decided our client was not a browser and served a challenge instead of the page. This is a verdict on the client, not on how much traffic we sent. Waiting, retrying, or slowing down will never clear it.

Unreachable shows as Unreachable · HTTP 503 and means the page genuinely did not load: the server errored, timed out, or refused the connection. Retrying later is reasonable here.

The distinction matters because bot protection is often reported with the status code 429, which normally means "too many requests". If you see a 429 with a bot protection badge, throttling your audits is not the fix.

Why nothing was scored

Every check reads the same page. If that page never arrived, the scores would describe the firewall's error page, not your site: no title, no headings, no structured data, thin content. Those would look like real findings and would be entirely wrong. SitePulse fails the audit instead.

Giving SitePulse access

All of this lives under Project → Settings → Crawl & Access, and needs the admin or owner role.

Let a browser render the page

Page rendering is set to Automatic by default. When bot protection refuses the ordinary request, SitePulse retries in a real browser, which passes the challenge the same way a visitor's browser would. This resolves most blocked sites on its own.

Set it to Always use a browser if a site blocks us consistently. It is slower and uses more resources, so it is not the default. Never disables the fallback if you would rather an audit fail fast.

Send an access header

For a site you control, the cleanest route is the access token your host already provides for automated testing. On Vercel, enable Protection Bypass for Automation and add its secret as a request header named x-vercel-protection-bypass. Staging passwords and firewall tokens work the same way.

Three things to know about these headers:

• They are sent only to this site's own host, plus any hosts you list under Also send to these hosts, and are dropped if a redirect leaves them. A token can never follow a link somewhere else.
• Values are encrypted and never shown again. After saving you will see "Saved, hidden". You can replace a value but not read it back.
• You must confirm you are authorised to send them before any header is used. They exist for sites you own or have written permission to test.

Sister domains behind the same protection

If the page links to sister sites with hreflang, such as a brand's country domains, SitePulse checks that each one links back. When those domains sit behind the same protection, the check can't see them and reports them as "couldn't verify". List them under Also send to these hosts to include them.

• List exact hostnames, like sefar.ch. That entry does not cover www.sefar.ch; list it separately if you need it.
• Only list hosts that accept the same token. On Vercel, a Protection Bypass secret works for every domain of the one project that issued it.
• Lighthouse still measures only the audited page's own host.

Ask the site's owner

If you do not control the site, the options are to have your address allowlisted in their firewall, or to have the challenge mode turned off. There is no setting in SitePulse that can substitute for that, and we do not try to disguise our requests to get around it.

A finding worth acting on

If a firewall challenges every non-browser client, it may also be challenging the crawlers you want to reach you, including AI assistants that cite sources. Worth checking with whoever configured it that search and AI crawlers are allowlisted, because robots.txt has no say here. Bot protection runs before robots.txt is ever read, so allowing a crawler there does not let it in.