Before an automated system can interpret a page, it has to receive a usable representation of that page. This is a delivery requirement, not a ranking strategy. Readability can remove one technical obstacle, but it cannot decide whether a crawler visits, whether an index retains the content, or whether a model uses it later.
That sequence sounds obvious until a browser hides the gap. A visitor may see a complete interface after scripts run, while a plain HTTP client receives an almost empty shell. A 200 response alone does not show whether the returned body contains the article, product details, documentation, or links the client needs.
Reading is a chain of separate conditions
It helps to split the problem into four questions:
- Can the client discover or otherwise obtain the URL?
- Does the server permit and complete the request?
- Does the response contain usable content, or can that client render what is missing?
- Can the client interpret the content well enough for its own task?
These questions are related, but a pass at one step does not answer the next. Google documents its own JavaScript pipeline as crawling, rendering, and indexing, with links extracted before and again after rendering. It also notes that an app-shell page may omit its actual content from the initial HTML (Google, "Understand the JavaScript SEO Basics"). That is evidence about Google Search's documented process, not a default capability shared by every automated client.
Vercel and MERJ reported a different result in 2024. Their primary data came from the Vercel network and nextjs.org, with two job-board sites used to compare observations across different stacks. In that sample, the listed OpenAI, Anthropic, Meta, ByteDance, and Perplexity crawlers did not execute JavaScript. The authors also reported that traffic they attributed to ChatGPT and Claude requested some JavaScript files without executing them (Zecchini et al.). The study supports inspecting initial HTML for the crawlers it measured. It does not establish permanent behavior for those services, cover every agent, or prove what any model does with fetched content.
Access policy and content delivery are not the same thing
The Robots Exclusion Protocol gives service owners a standard way to publish rules that crawlers are requested to honor. RFC 9309 is equally clear that those rules are not access authorization (Koster et al., sec. 1). A permissive robots.txt file therefore does not prove that a request will succeed. A firewall, authentication layer, rate limit, challenge, redirect loop, or server error can still prevent retrieval.
Technical fetchability can also conflict with the published policy. A client may be able to retrieve a URL that the site's crawler policy asks it not to crawl. A readiness review should report both the observed response and the stated directive rather than collapse them into a single label such as "accessible."
Discovery signals have similar limits. Google says submitting a sitemap is only a hint and does not guarantee that Google will download it or use it for crawling (Google, "Build and Submit a Sitemap"). Structured data can label facts in a standardized form, but even correct markup does not guarantee a Google rich result (Google, "General Structured Data Guidelines"). Those documents can make a site's published intent easier to inspect. They do not control later selection.
A practical reading test
Start with one public URL that matters. Fetch it without a browser and save the final status, response headers, and response body. Compare the body with the page a visitor sees after rendering. Look for the primary text, a descriptive title, ordinary crawlable links, and any metadata that the page is supposed to publish.
If important content is absent, identify where it appears. It may arrive in the initial HTML, embedded application data, a later API request, or client-generated DOM. That location is an observation. Whether to change the delivery architecture depends on the clients the site intends to support, along with performance, security, and maintenance constraints.
Next, inspect robots.txt, page-level robots directives, the canonical URL, relevant structured data, and discovery files. Check whether they agree with the visible page and with the site's actual policy. Do not add a file solely to collect a green check. An accurate absence is better than machine-readable metadata that points to stale, private, or unsupported resources.
MachineRead applies this sequence through specific public checks. Its bot-access check sends tracked bot user-agent requests and compares their status, routing, and basic content characteristics with a comparison baseline, normally an attempted browser-identifying request. The broader methodology keeps directives, raw-HTML readability, schema, and discovery evidence in separate findings. Remediation can then follow the observed condition instead of a broad claim about "agent visibility."
What this check can and cannot establish
The check can establish what MachineRead's tracked user-agent requests received at a specific time and whether their status, routing, or basic content characteristics differed from its comparison baseline. Related checks can identify published directives, parseable metadata, and reachable discovery files.
It cannot establish that every crawler receives the same response, that a named provider rendered or retained the page, or that a model will retrieve, cite, rank, recommend, or act on the content. It also cannot infer intent from a block or prove content quality from successful parsing.
Works Cited
Google. "Build and Submit a Sitemap." Google Search Central, updated 8 July 2026, https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap. Accessed 7 Aug. 2026.
Google. "General Structured Data Guidelines." Google Search Central, updated 10 July 2026, https://developers.google.com/search/docs/appearance/structured-data/sd-policies. Accessed 7 Aug. 2026.
Google. "Understand the JavaScript SEO Basics." Google Search Central, updated 4 Mar. 2026, https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics. Accessed 7 Aug. 2026.
Koster, Martijn, et al. "Robots Exclusion Protocol." RFC 9309, Internet Engineering Task Force, Sept. 2022, RFC Editor, https://www.rfc-editor.org/rfc/rfc9309.html. Accessed 7 Aug. 2026.
Zecchini, Giacomo, et al. "The Rise of the AI Crawler." Vercel, 17 Dec. 2024, https://vercel.com/blog/the-rise-of-the-ai-crawler. Accessed 7 Aug. 2026.
See also
- Launch post - what the audit measures and does not measure
- Methodology reference - the 13 check groups and probe definitions
- Related: bot access and schema.org JSON-LD