robots.txt is usually filed under search hygiene, a small file that tells Googlebot what to skip. That description was accurate for twenty years. It stopped being complete around 2023, when crawlers arrived that fetch pages to answer questions, not to build an index. The file is now where your AI policy is written, whether or not anyone at your organization has decided what that policy is.
The blocking data makes the shift concrete. Roughly 5.6 million websites had added OpenAI's GPTBot to a disallow list by late 2025, up from about 3.3 million at the start of that July, an increase of nearly 70% in a few months (The Register). Among top news sites the posture is close to uniform, with about 79% blocking AI training bots (BuzzStream). GPTBot is the single most-disallowed AI crawler in the rules that MachineRead and others parse.
The more interesting number isn't how many sites block. It's how many chose to. Cloudflare introduced managed robots.txt controls for AI training on July 1, 2025, making it easier for publishers to opt in to AI-crawler blocking (Cloudflare). That means a large number of sites now ship a restrictive AI policy that arrived with the hosting, not with a decision, and the site owner may hold the opposite view without knowing the file disagrees. This cuts both ways. Blocking AI crawlers is a legitimate position, particularly for publishers whose content is the product. The audit doesn't score a deliberate block as a failure of judgment. What it flags is the mismatch between the policy you believe you have and the one your file states.
These crawlers aren't interchangeable, and one directive doesn't cover them. Training crawlers, search-index crawlers, and live retrieval fetchers can be separate user agents from the same vendor. A rule aimed at training can cut off the retrieval path that would have cited you. MachineRead checks Allow and Disallow directives across 15 tracked AI bots for this reason, distinguishing an explicit grant from silence. Under the protocol, silence means permitted. The audit treats an unnamed bot as allowed but scores it below an explicit grant, because silence records no intent and tells you nothing about what happens when the default changes.
This check group carries 6 of the 56 points, the heaviest single row in the AI access pillar. That weight reflects reach rather than difficulty. Editing this file is a two-minute job, and it can silently determine whether an entire category of crawler ever reaches the rest of your site.
Here's what I'd check first. Open your robots.txt and read it as a policy statement, not configuration. Check whether it says what you'd say if asked. If you want retrieval traffic, name the major AI user agents and allow them explicitly. Be aware that a hosting default may have decided otherwise. If you want to block training while staying citable, separate training crawlers from retrieval ones. Don't write one blanket rule. What matters is that the position in the file is one you actually hold. This is the rare setting where doing nothing is itself a choice somebody else may have made for you.
Sources
- The Register: Publishers say no to AI scrapers (2025-12-08): Secondary source
- BuzzStream: Publishers Block AI Study (2025-10-01): Secondary source
- Cloudflare: Control Content Use for AI Training (2025-07-01): Primary source
See also
- Methodology reference - documents the robots_txt check group and its 6 checked points
- Launch post - covers how AI-bot directives fit the AI access pillar
- Related: bot access and sitemap