A page can work perfectly in a browser and still give a very different answer to a simple HTTP request. That difference matters when a site depends on client-side rendering, sends a challenge page to unfamiliar clients, or delivers an empty application shell before JavaScript runs.
MachineRead's bot-access check stays deliberately narrow. It attempts a browser-identifying baseline request and otherwise reuses the initial audit fetch as its comparison baseline. It then makes a paced set of requests carrying tracked crawler user-agent strings and compares what they receive at audit time. A passing result does not certify access for an official crawler, whose network identity or treatment may differ from MachineRead's probe.
The browser is not a useful stand-in for every client
Google documents a rendering system that can process JavaScript before indexing content (Google). That capability is real, but it is specific to Google's crawler stack. It should not be used as a proxy for every system that fetches a URL.
A Vercel and MERJ study published in 2024 reported no JavaScript rendering by the named OpenAI, Anthropic, Meta, ByteDance, and Perplexity crawlers in its tests (Zecchini et al.). It separately observed traffic the authors attributed to ChatGPT and Claude fetching some JavaScript resources without executing them, and noted that content included in the initial response could still be available (Zecchini et al.). This is a time-bound observation about the clients and sites in that study, not a rule for automated retrieval generally.
That is why the check starts with the raw response. It can be accessible and still be unhelpful if it contains only a small root element and script references. Conversely, a page may be fully usable to a browser-rendering crawler while a plain HTTP client sees little beyond the shell. Neither observation establishes what every crawler, search engine, or model will do next.
What the check observes
The probe uses plain HTTP GET requests with crawler-identifying user-agent strings. Its comparison baseline comes from an attempted browser-identifying request or, if that request fails, the initial audit fetch. It does not execute JavaScript, run a browser renderer, or solve a challenge. It retries an HTTP 429 response once after a delay, but it does not fall back to a browser for the crawler probes.
The result combines several observations:
- Did a probe end in a fetch error, selected blocking status, or recognized challenge response?
- Was the visible text much thinner than the comparison baseline?
- Did the status, final URL, canonical URL, response size, or visible word count differ materially from that baseline?
A denial, challenge page, persistent rate limit, or thin initial document gives a site owner a specific condition to inspect. It does not mean automated access is categorically wrong. Some sites intentionally restrict it. The question is whether the observed behavior matches the site's publication and discovery goals.
What this check can and cannot establish
The check can establish what MachineRead received for its comparison baseline and tracked user-agent requests at one point in time. It can flag selected response failures, challenge fingerprints, thin content, and routing differences under those conditions.
It cannot authenticate an official crawler from a user-agent string, reproduce provider-specific IP reputation or request history, or show that another client will receive the same response. It also cannot establish indexing, training use, ranking, citation, referral, or model behavior.
How to use a finding
When the check reports limited content, fetch the affected URL with a non-browser user agent and inspect the response headers and opening HTML. If critical public text appears only after JavaScript runs, decide whether that is appropriate for the audience you want to reach. If a security control blocks the request, confirm whether the policy is intentional.
Record whether the behavior is intentional, then repeat the request after any rendering, hosting, or security change. The evidence remains the response MachineRead received, not a forecast about a crawler's next action.
Works Cited
Google. "Understand the JavaScript SEO Basics." Google Search Central, updated 4 Mar. 2026, https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics. Accessed 7 Aug. 2026.
Zecchini, Giacomo, et al. "The Rise of the AI Crawler." Vercel, 17 Dec. 2024, https://vercel.com/blog/the-rise-of-the-ai-crawler. Accessed 7 Aug. 2026.
See also
- Methodology reference - documents the bot-access check group
- Launch post - explains the audit's evidence boundaries
- Related: robots.txt and why agents need to read your site