Blog

Indexing directives are not visibility outcomes

Find accidental noindex rules and restrictive snippet settings in page HTML and response headers, and distinguish them from deliberate publishing choices.

author: George Jieh
published: 2026-09-25
reading: 4 min

A public page can look ready to share while its response still says noindex. The instruction may have come from a staging template, a shared layout, or a response header that never appears in the visible page.

MachineRead checks the returned HTML and headers for selected indexing and preview restrictions. The useful result is the instruction and where it was found. Whether that instruction is a mistake depends on what the publisher intends the page to do.

Check indexing instructions alongside crawl policy

Google documents noindex as a rule asking that a page, media, or resource not appear in search results. Without it, the resource may be indexed; inclusion is not automatic (Google Search Central, "Robots Meta Tag, data-nosnippet, and X-Robots-Tag").

Keep this separate from robots.txt. Crawl policy controls which URLs a crawler is asked to access, while page-level indexing instructions address the treatment of a resource. Inspect both when diagnosing a public page, rather than assuming a permissive robots.txt file rules out an accidental noindex (Google Search Central, "Robots.txt Introduction and Guide").

Snippet controls change previews, not index proof

Snippet and preview controls are more specific than a broad indexing block. Google documents nosnippet as preventing a text snippet or video preview in search results for the page. It also documents data-nosnippet as a way to mark textual parts of an HTML page that should not be used as a snippet (Google Search Central, "Robots Meta Tag, data-nosnippet, and X-Robots-Tag").

Those controls can be deliberate. A site may want certain text excluded from previews, or may want a stricter image or video preview policy. The risk is treating the presence of a snippet control as a complete diagnosis. A restrictive preview instruction does not prove that Google selected a snippet, that the page was shown, or that another system will honor the same boundary.

A useful audit therefore separates three states:

Blocking directives such as noindex or none.
Preview and quote restrictions such as nosnippet, data-nosnippet, max-snippet:0, max-image-preview:none, noimageindex, or max-video-preview:0.
A clean fetched homepage response with no selected blocking or restrictive signals under the check.

Only the third state means the check did not find those selected obstacles on the fetched homepage. It does not mean every URL on the site is indexable.

What MachineRead observes

MachineRead parses the fetched homepage HTML and the X-Robots-Tag response header. The check recognizes generic meta robots directives, selected crawler-specific meta tags, and crawler-scoped X-Robots-Tag directives. It then applies its five-point indexing-directives rubric.

Blocking signals win first. If the homepage contains noindex or none, the check reports fail at 0/5. If the page allows indexing but restricts snippets, images, or previews through the selected signals, the check reports a partial result. It counts data-nosnippet sections, treats max-image-preview:standard as a large-image-preview limitation, and reports a pass at 5/5 only when the selected blocking and restrictive signals are absent.

The score reports the directives found in that response. To determine whether Google actually indexed the page or displayed a preview, you need evidence from Google rather than another inspection of the markup.

How to use the finding

Start with the returned page, not the browser screenshot. Inspect the homepage response headers for X-Robots-Tag. Then inspect the static HTML for meta name="robots", crawler-specific meta tags, and data-nosnippet attributes.

If a public homepage is supposed to appear in search, remove accidental noindex or none directives from that page and from any shared layout that emits them. If the finding points to preview restrictions, decide whether they are intentional. A privacy, licensing, or brand-control reason may be valid. If snippets and previews are wanted, narrow or remove the restrictive directive that is being applied too broadly.

Then retest the deployed page. Templates, middleware, redirects, and edge headers can change what a plain request receives. The evidence that matters for this check is the response MachineRead can fetch at audit time.

Works Cited

Google Search Central. "Robots Meta Tag, data-nosnippet, and X-Robots-Tag." Google for Developers, https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag. Accessed 9 Sept. 2026.

Google Search Central. "Robots.txt Introduction and Guide." Google for Developers, https://developers.google.com/search/docs/crawling-indexing/robots/intro. Accessed 9 Sept. 2026.