llms.txt isn’t an AI-visibility strategy. Here’s what it can do.

llms.txt is commonly pitched as a way to improve search and AI visibility. Yoast reinforces that impression with a one-click generator for guiding assistants such as ChatGPT to a site's important content. Packaged inside an SEO product, it looks like another tool for improving discovery. But llms.txt has a different and much narrower use.

What the evidence says

llms.txt is a plain text file that points AI systems to your most important pages, like a curated table of contents for a machine. It usually sits at a site's root, though it can also live at a subpath like /docs/llms.txt to cover just the pages beneath it. The spec is deliberately bare. Some documentation platforms also publish an llms-full.txt that concatenates the linked content into a single file.

The belief that the file lifts your standing in search and AI answers spread more by repetition than by evidence, helped along by a wave of generative engine optimization (GEO) guides that list it beside schema as a ranking tactic. The primary sources say otherwise. Google's own documentation states that you don't need to create AI text files or machine-readable markup to appear in its AI features, and that its best practices for AI search are the same as for regular Search.

Other AI systems have made no such commitment either; no major search or answer provider has publicly identified llms.txt as a ranking or citation signal. And the usage data is telling. An Ahrefs study of 137,210 domains found that in May 2026, 97 percent of published llms.txt files received no requests at all, and among the few that did, AI retrieval bots accounted for only about 1 percent. Ahrefs is careful to note that a request would not even prove the file was read, only that something asked for it. On the evidence available, llms.txt is not a visibility lever.

What actually helps for Google's AI search

For Google's generative search features, familiar SEO fundamentals still apply. Clear technical structure makes a site crawlable, and structured data can make content eligible for rich results. Well-scoped pages and genuinely useful content are what get surfaced. Google's guidance is explicit that optimizing for generative AI search is still, at its core, SEO.

Documentation structure matters here because retrieval happens at the page and passage level. An agent has a better chance of landing on the right answer when page titles are specific and outdated material is clearly marked. A generator can assemble an llms.txt from your sitemap, but the sitemap alone cannot tell it whether to drop a deprecated API reference or feature a migration guide. Someone who understands the product still has to decide what is current and what deserves emphasis. That editorial oversight is part of making documentation AI-ready.

The clearest use case: coding agents

llms.txt has an active audience in coding agents. Claude Code is one documented example: Anthropic publishes an llms.txt index for its own developer docs, and the Ahrefs data shows Claude Code among the top requesters of these files. The use case is broader than one product, though, since any agent with web access can use the file when a developer or a documentation page directs it there. When pointed to the file, a coding agent can use its descriptions to pick the relevant API references or tutorials without loading the entire documentation set before it writes code.

The catch is discovery. Ahrefs data showing agents requesting these files also shows retrieval bots rarely doing so, and no AI bot probing for the file on sites where it doesn't exist. Agents generally reach for an llms.txt only after a link, an index, or a user points them to it. It behaves like an agent-friendly documentation index that someone has to hand over, not a standard index every assistant seeks out on its own. That is why the file suits a particular profile: if you publish an API or SDK that developers build against with AI assistance, you have a plausible audience for it today. Outside that case, there is little evidence it does much.

How to build a useful llms.txt

A few things make the difference between a file agents use and one that adds nothing beyond your sitemap.

Keep it selective. If a developer asks an agent how to authenticate, the index should lead to the authentication guide, not every page that mentions tokens. A short description beside each link helps the agent choose without opening everything.

Make it discoverable. An agent may never try /llms.txt unless something points there. The v2 proposal recommends a rel="describedby" link or an equivalent HTTP Link header, so if a platform generates the file for you, check whether it adds that too.

Keep it current. A file that has drifted from the product will point an agent confidently at the wrong answer.

Should you publish one?

If you publish public API or SDK documentation and can maintain the file cheaply, llms.txt is a reasonable experiment. Curate its links, make it discoverable, and check your server logs to see whether agents request it. If that use case does not apply, improve the underlying documentation first.

Talk to us about documentation developers and AI agents can navigate.