Blog

Signals that help crawlers find updated public content

Search discovery checks public signals related to reachability, initial HTML content, and declared modification dates.

author: George Jieh
published: 2026-08-04
reading: 6 min

Search discovery is best treated as a path audit. It asks whether a site exposes routes to public URLs, whether sampled requests retrieve useful responses, and whether it publishes freshness cues that MachineRead can parse. Those are observable conditions. Crawling, rendering, indexing, ranking, traffic, citations, and agent use all happen later, under systems the publisher does not control.

MachineRead tests that public chain without turning its links into a forecast.

Start with routes a client can see

A page can be exposed through ordinary site navigation, a sitemap, or both. For Google specifically, the dependable HTML pattern for a crawlable link is an a element with an href that resolves to a web address; Google says most other link formats are not parsed and extracted reliably (Google, "Link Best Practices for Google").

A sitemap is another route into the site, but it is a declaration, not proof of an outcome. Google says that a sitemap can help its systems discover URLs while explicitly warning that listed items are not guaranteed to be crawled or indexed (Google, "Learn about Sitemaps"). This makes a sitemap useful evidence of what a publisher has exposed, but not evidence of what any search engine has done with it.

The same restraint applies to robots.txt. The Robots Exclusion Protocol lets a service owner publish rules that crawlers are requested to honor, and the standard states that those rules are not access authorization (Koster et al., sec. 1). Reading a rule can establish the site's declared policy for a named crawler. It cannot establish whether that crawler requested the page, how it interpreted every other signal, or whether an unrelated client followed the rule.

The response matters as much as the route

A discovered URL still has to return something useful. MachineRead fetches selected pages over HTTP and examines the response without running a browser. That gives the audit the returned status, final URL, selected response headers, and initial HTML under that retrieval condition.

Initial HTML and rendered content are not interchangeable. Google's own documentation describes an app-shell case in which the first response lacks the page content and Google later executes JavaScript to produce rendered HTML. The same documentation cautions that not all bots can run JavaScript (Google, "Understand the JavaScript SEO Basics"). Google's rendering capability is therefore not a sound assumption about another search crawler, archival tool, or agent.

MachineRead therefore samples what is present before client-side execution. On sampled sitemap pages, it checks titles, canonical links, robots directives, parseable JSON-LD, and extractable text in the initial response. Separate heuristics inspect target-page anchors and sitemap URL strings for public trust pages, and validate hreflang only where it is published. These observations do not prove that another system will use the signals. An absent signal identifies a gap in the raw document, but determining whether browser rendering supplies it requires a separate test.

Dates should report changes, not deployments

Freshness cues are useful only when they describe the content. Google's sitemap guidance says it uses lastmod when the values are consistently and verifiably accurate, and that the field should reflect the last significant update to the page rather than a routine copyright change (Google, "Build and Submit a Sitemap"). By that standard, a build that stamps every URL with today's date may produce valid XML while weakening the meaning of the date.

MachineRead inspects sitemap lastmod coverage, syntax, and future dates. It does not compare a declared date with the page's revision history. The fallback heuristic can also observe a linked publishing section, dated target-page metadata, or a reachable, nonempty RSS, Atom, or JSON feed. It reports feed date coverage and recency when dates are present. These alternatives describe different publishing practices. A brochure site may have no reason to maintain a feed, while a publication may use one as a compact record of recent URLs. In either case, the audit records what is exposed; it does not infer a crawl schedule from it.

What MachineRead records

The check brings several observations into one report row:

whether the published robots.txt rules block Googlebot or Bingbot from the audited target path under the URL constructed by the check;
whether a valid sitemap is found at the conventional path or through a robots.txt reference, and whether sampled entries use coherent same-site HTTPS URLs;
whether selected sitemap URLs return accessible responses whose initial HTML exposes useful text, titles, canonical links, robots directives, and parseable JSON-LD; the normal shared-evidence path also applies a local title-and-description coherence heuristic;
whether at least 80 percent of sampled sitemap entries declare lastmod, with no invalid or future values encountered in the parsed sitemap documents, or the target page exposes a publishing-section link, dated metadata, or a nonempty parseable feed; and
whether target-page anchors or sitemap URLs expose About, Contact, and Privacy routes, with contextual findings for other policy pages and additional checks for hreflang only when a site publishes it.

The sample is intentionally limited. MachineRead does not crawl every URL, execute JavaScript, query a search-result page, inspect Search Console, or call a paid indexing provider. A full score means the four point-bearing conditions passed and no applicable hreflang, core trust-page, or commerce-policy issue capped the row. Optional findings and sample caveats still require review. A weak result identifies a response or declaration to inspect; it does not diagnose a search-performance problem.

Review the chain in the same order. Request robots.txt and the sitemap directly. Open several listed URLs with JavaScript disabled or inspect their raw responses. Confirm that important pages have ordinary links from navigation or relevant content. Compare lastmod and feed dates with actual editorial changes. If those checks disagree, repair the declaration or the page rather than trying to predict how a crawler will compensate.

What this check can and cannot establish

It can establish which HTTP responses, HTML elements, links, directives, sitemap entries, and declared dates MachineRead observed in its limited sample at audit time. It can also identify contradictions, such as a sitemap URL that fails to load or a modification date that does not parse.

It cannot establish that a named search engine or agent discovered the URL, executed its scripts, crawled it again, indexed it, selected it for a result, ranked it, sent traffic, cited it, or used it in a task. It also cannot prove that a syntactically valid lastmod value corresponds to an actual content change. Those outcomes and historical claims require evidence beyond this check.

Works Cited

Google. "Build and Submit a Sitemap." Google Search Central, updated 8 July 2026, https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap. Accessed 7 Aug. 2026.

Google. "Learn about Sitemaps." Google Search Central, updated 10 Dec. 2025, https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview. Accessed 7 Aug. 2026.

Google. "Link Best Practices for Google." Google Search Central, updated 10 Dec. 2025, https://developers.google.com/search/docs/crawling-indexing/links-crawlable. Accessed 7 Aug. 2026.

Google. "Understand the JavaScript SEO Basics." Google Search Central, updated 4 Mar. 2026, https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics. Accessed 7 Aug. 2026.

Koster, Martijn, et al. "Robots Exclusion Protocol." RFC 9309, Internet Engineering Task Force, Sept. 2022, RFC Editor, https://www.rfc-editor.org/rfc/rfc9309.html. Accessed 7 Aug. 2026.