A useful robots.txt file makes a site's crawl policy legible. It does not have to be permissive. Its job is to tell crawlers, through named groups and URL-path rules, what they are asked to access or avoid (Koster et al., secs. 2.1-2.2.2). That makes the file a public policy signal, not a gate and not a forecast of what happens after a URL is discovered.
Policy is not enforcement
RFC 9309 describes the Robots Exclusion Protocol as rules that crawlers are requested to honor and states directly that those rules are not access authorization (Koster et al., sec. 1). A Disallow rule therefore should not be treated as protection for private material. The RFC warns that paths listed in robots.txt become publicly discoverable and recommends real application-layer security when access must be controlled (Koster et al., sec. 3).
An Allow rule does not reverse that fact. Because the protocol is not authorization, the rule says nothing about whether an HTTP request will clear a login or another access control (Koster et al., secs. 1, 3). MachineRead therefore reports robots.txt policy separately from non-browser fetch results.
Read the applicable group, not a keyword count
A robots.txt group begins with one or more User-agent lines followed by Allow or Disallow rules. A crawler first looks for a group matching its product token; the * group applies when no product-token group matches. If neither a matching group nor a wildcard group exists, no rules apply (Koster et al., secs. 2.1-2.2.1).
Path evaluation also depends on specificity. When more than one rule matches a URL, RFC 9309 says to use the rule with the longest matching path. If equally specific Allow and Disallow rules conflict, Allow should win (Koster et al., sec. 2.2.2). Reading a file by searching for a bot name or counting Disallow lines can therefore miss the policy that applies to the requested URL.
Under RFC 9309, crawlers set their own product token, which should appear within the identification string they send to the service (Koster et al., sec. 2.2.1). A correctly matched rule therefore establishes the site's published instruction for that token. The file does not authenticate a requester, demonstrate compliance, or identify why a crawler wants the page.
What MachineRead observes
MachineRead fetches the public /robots.txt file and evaluates the audited target path under the URL it constructs for a maintained set of crawler product tokens. Its parser classifies each tracked token through an explicit group, a wildcard group, or no named rule, and it reports selected file-quality conditions such as malformed directive lines and non-absolute or non-HTTP(S) Sitemap directives.
The result describes one file at audit time. It can reveal a blanket rule that catches more clients than intended, an explicit crawler rule that differs from the wildcard policy, or syntax that deserves review. A restrictive result may be exactly what the publisher chose. MachineRead surfaces the difference; it does not decide the publisher's policy.
What this check can and cannot establish
The check can establish whether MachineRead retrieved a robots.txt file, how its parser classified the audited target path for the crawler tokens it tracks, and which selected structural issues appeared in that response.
It cannot establish that a requester is who it claims to be, that any crawler honored the rules, that the same client can fetch other paths, or that content was indexed, used for training, quoted, cited, or ranked. It also cannot turn robots.txt into authorization or a security control.
Maintain the policy you mean to publish
Start with the wildcard group, then inspect each crawler-specific exception and test representative public paths against the resulting rules. Remove stale groups when the policy changes. Keep sensitive route names out of the file and protect those routes with authentication or another real access control (Koster et al., sec. 3).
After a hosting, CDN, or bot-management change, compare the published rules with an actual non-browser response. That pair of checks answers two useful but separate questions: what the site asks crawlers to do, and what the site serves when one arrives.
Works Cited
Koster, Martijn, et al. "Robots Exclusion Protocol." RFC 9309, Internet Engineering Task Force, Sept. 2022, RFC Editor, https://www.rfc-editor.org/rfc/rfc9309.html. Accessed 7 Aug. 2026.
See also
- Methodology reference - documents the robots_txt check group
- Launch post - explains the audit's evidence boundaries
- Related: bot access and sitemap