The bot access probe asks one question: when something that is not a browser requests this page, does it get the content? Most teams assume bot access is just uptime: if the site loads in Chrome, it's available to everything. That assumption conflates two different things, and the gap between them is where most invisible failures live.
Two changes make that assumption worth rechecking. The first is deliberate blocking. Cloudflare introduced AI-crawler blocking controls on July 1, 2025, encouraging every new site to opt in (per Cloudflare's own blog post), and Cloudflare's customers blocked hundreds of billions of AI bot scraping requests in the months following (The Register reports 416 billion) (The Register). A great many sites are now refusing AI crawlers by default rather than by decision, which is fine if it was a decision and expensive if nobody made one. The second is rendering. Vercel and MERJ measured roughly 500 million AI crawler fetches and found zero JavaScript execution (Vercel), so a client-rendered page can return a 200 and still hand back nothing usable.
Those two failure modes look different in the logs but produce the same outcome. A WAF challenge, a 403, or an aggressive rate limit stops the fetch outright. A client-rendered shell completes the fetch and delivers an empty document. In both cases the agent ends up with no content. That is why the probe checks for both a successful status and substantive HTML, not just the status code.
The probe deliberately behaves like the average retrieval client rather than like a browser. It issues a plain HTTP GET with a user agent that identifies it as a bot, and it does not execute JavaScript, render a DOM, solve challenges, or retry behind a headless browser. That restraint is deliberate. Measuring with a full browser would show you what Googlebot sees (Googlebot renders JS) and would hide the failure that affects crawlers that don't.
Bot access carries 6 of the 56 checked points, which understates it. I'd rather say that directly than quietly weight it higher. Its real weight is structural. The other twelve check groups are all conditional on this one, since structured data cannot be extracted from a page that was never delivered, and a sitemap cannot help an agent that is being turned away at the door. A site that fails this probe does not lose 6 points in any meaningful sense. It fails the audit, and the remaining score describes a document no agent will ever read.
Here's what I'd do today. Test it yourself before trusting any dashboard. Run curl against your homepage with a bot user agent and look at what actually comes back. If you get a 200 with real HTML, you pass. If you get a challenge page, a 403, or a shell with no content, you've found the finding that outranks everything else. The fix depends on which failure you hit: adjust WAF rules to let legitimate crawlers through, or move to server-side rendering so the initial HTML carries the content. Fix this first. Every other improvement sits behind it.
Sources
- The Register: Publishers say no to AI scrapers (2025-12-08): Secondary source
- Vercel: The Rise of the AI Crawler (2025-10-01): Primary source
See also
- Methodology reference - documents the bot_access check group and its structural role
- Launch post - covers how bot access fits the AI access pillar
- Related: robots.txt and why agents need to read your site