MR
MachineReadAI & search readiness audit

Blog

If a Crawler Can't Find It, It Doesn't Exist

Search discovery is the check group that asks whether new and updated content can actually be found. It's usually framed as a freshness question, with the advice that publishing often improves rankings. Recency does matter for some queries. But framing freshness as a ranking input skips the more basic function: a last-modified date is a re-fetch trigger. It's how a crawler decides which of your ten thousand pages are worth requesting again this week. A site that can't answer that question credibly gets crawled on a schedule that has nothing to do with when it changed.

Three things have to hold for discovery to work, and they fail independently. The content has to be indexable, which is a robots and meta-directive question. It has to be present in the initial HTML response, which is a rendering question. And there has to be a surface that announces new content, which is a feed or index question.

The rendering condition is the one that has changed most and gets checked least. The crawlers behind most AI retrieval don't run JavaScript at all (the measurements are covered in the bot access post). For those crawlers a client-rendered page is an empty page regardless of how fresh it is. A site can publish daily, maintain a perfect sitemap, and still present nothing to retrieve because the content arrives after a script the crawler never runs. Google renders JavaScript and Gemini inherits that capability. That's precisely why this failure is so easy to miss: the one crawler most teams monitor is the one that doesn't hit it.

The freshness condition fails differently, through a signal that's technically present and substantively false. Build-time timestamps that update every page on every deploy are the common case. Once a crawler concludes your dates are noise, it stops using them and falls back to its own re-crawl heuristics. The cost isn't a penalty. It's a lost channel. You no longer have a way to say that one specific page changed and deserves another look.

The third condition is the one most sites simply skip. A feed, whether RSS or Atom, is a standing announcement surface that tells anything subscribed to it what is new, without requiring a full re-crawl. Feeds are unfashionable and they remain the cheapest mechanism for the purpose. MachineRead checks for one under the blog and content presets and doesn't penalize sites where the format would be meaningless.

The search_discovery group carries 4 of the 56 checked points. The sitemap validated under the indexing group feeds it, since the dates there are the freshness evidence this check evaluates. The scoring is deliberately conditional on the preset you pick. A corporate brochure site with no publishing cadence shouldn't be marked down for lacking a feed it has no use for.

Here's what I'd verify, in order. First, confirm your content exists in the HTML before JavaScript runs. Fetch the page with curl and search the response for a sentence you can see on screen. If it's missing, that's the finding. Server-side rendering or prerendering is the fix. Then make lastmod truthful, even if that means fewer pages carry a date. Publish a feed if you publish anything on a cadence. None of this is a growth tactic. It's the difference between a site that can tell machines what changed and one that has to wait to be noticed.

See also