Blog

llms.txt is optional, not a visibility shortcut

llms.txt is a proposed discovery file for language-model tools. A MachineRead result describes the response and basic shape, not client support or use.

author: George Jieh
published: 2026-07-27
reading: 4 min

llms.txt is a publishing proposal (Howard). It is not a visibility shortcut. A site owner can use it as a compact Markdown guide to important public material for tools that choose to read it (Howard). The file does not show that any particular crawler requested it, understood it, or used its links.

The proposal was published by Jeremy Howard on 3 September 2024 and remains open to community input (Howard). Publishing a conforming file does not by itself demonstrate provider support.

What the proposal defines

The proposed root location is /llms.txt, although the format also permits a subpath (Howard). The document is Markdown, begins with an H1 naming the site or project, and may continue with a short blockquote summary, explanatory text, and H2 sections containing lists of links (Howard). An H2 named Optional marks secondary links that can be omitted when a shorter context is needed (Howard).

The proposal also suggests offering clean Markdown versions of useful pages and leaves the processing of llms.txt to each application (Howard). The format stops there: it tells a publisher how to organize a file but does not require any remote system to fetch or act on it.

A useful llms.txt is therefore closer to a curated documentation index than a second copy of the site. It can name the product, explain what the linked material covers, and direct a supporting client to canonical documentation. The linked pages still need to be accurate and independently accessible.

Keep discovery separate from policy

The proposal distinguishes llms.txt from both sitemaps and robots.txt: it presents the file as a curated overview, while those other files serve different discovery and crawler-policy purposes (Howard). The Robots Exclusion Protocol defines rules that crawlers are requested to honor, and it explicitly states that those rules are not access authorization (Koster et al., sec. 1). llms.txt is not an alternative crawler-control or security mechanism.

Keeping those roles separate avoids two common mistakes. Do not put sensitive links in llms.txt on the theory that only selected tools will read it. Treat the file as public. Do not move essential policy or documentation into the file while leaving the ordinary site incomplete. Keep public pages and navigation complete, and maintain other discovery files for the clients that use them.

A maintenance test is more useful than a publication claim

Before publishing, decide what job the file serves. Documentation-heavy sites might use it to point to an overview, tutorials, API references, and change logs. A smaller brochure site may have no need for an extra index.

For a file you do publish:

Use one clear H1 and a short factual summary.
Link to canonical public URLs rather than temporary exports or private environments.
Describe why each link is useful instead of repeating keywords.
Put secondary material in the Optional section.
Check links after documentation moves and remove stale entries.
Keep the file consistent with the facts readers see on the linked pages.

MachineRead evaluates llms.txt as part of a broader LLM Text and Markdown Access check. It looks at the response and basic document shape alongside sitemap and Markdown-access evidence. The result reports which candidate text and discovery responses were returned during the audit and met its basic checks. It does not test a provider's production crawler or model.

What this check can and cannot establish

The check can establish whether the audited site returned a candidate /llms.txt response under the tested conditions and whether that response starts with a hash character (#) under its basic shape heuristic and contains a URL or Markdown link. The broader check can also report related sitemap and Markdown-access observations.

It cannot establish that a named crawler supports the proposal, that the file was consumed, or that its links influenced retrieval. It cannot predict ranking, traffic, citations, model answers, or referral behavior. It also cannot turn public descriptions into access control or verify every factual statement on every linked page.

Works Cited

Howard, Jeremy. "The /llms.txt File." llms-txt, 3 Sept. 2024, https://llmstxt.org/. Accessed 7 Aug. 2026.

Koster, Martijn, et al. "Robots Exclusion Protocol." RFC 9309, Internet Engineering Task Force, Sept. 2022, RFC Editor, https://www.rfc-editor.org/rfc/rfc9309.html. Accessed 7 Aug. 2026.