llms.txt (a Markdown guide for AI agents)
llms.txt is a proposed plain-text file at /llms.txt that gives language models and AI agents a short, curated overview of a site: a title, a summary and lists of links to the pages that matter, written in Markdown. It is a community proposal, not a web standard, and Google has not announced that Search uses it. It costs little to add and does not affect indexing either way.
What the proposal says
The llms.txt specification proposes adding a /llms.txt Markdown file to websites to provide LLM-friendly content. The reasoning: HTML wraps information in navigation, ads and scripts, and context windows are too small for whole sites. A short, curated file lets an agent find the right page and fetch only that.
The file can sit at the root or at any path, in which case it covers the pages under that path; where several apply, the most specific one wins. The proposal also recommends serving a clean Markdown version of important pages at the same URL with .md appended, and pointing to it with rel="alternate" type="text/markdown" and to the llms.txt file with rel="describedby", as HTML <link> elements or an HTTP Link: header.
How it looks
The format is deliberately simple. According to the specification, a file contains, in this order:
- An H1 with the name of the project or site. This is the only required section.
- A blockquote with a short summary containing the key information needed to understand the rest.
- Zero or more Markdown sections without headings, with more detail on the project and how to read the files.
- Zero or more sections delimited by H2 headings containing "file lists": Markdown lists where each item is a link
[name](url), optionally followed by a colon and notes.
By convention, a final ## Optional section holds secondary links an agent can skip when context is short. A minimal file for a small shop could read:
# Northwind Bikes
> Online bike shop based in Leeds. Ships to the UK and Ireland. Pages below cover products, sizing, delivery and returns.
## Products
- [City bikes](https://example.com/city-bikes/): current models with prices and stock
- [Sizing guide](https://example.com/sizing/): frame size by rider height
## Policies
- [Delivery](https://example.com/delivery/): costs, lead times and tracking
- [Returns](https://example.com/returns/): 30-day returns, how to start one
Each line in the file is plain Markdown; the server should return it as text/plain or text/markdown with a 200 status, not as an HTML page.
How Google handles it
There is nothing to report. Google's overview of its crawlers and its list of common crawlers do not mention llms.txt, and no Google Search documentation describes reading it. The file is not a ranking or indexing signal and it is not a substitute for a sitemap.
What Google does document is a robots.txt token for AI use. Per the common crawlers page, Google-Extended is a standalone product token that publishers can use in robots.txt to manage whether content Google crawls may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI. It has no separate user agent string; crawling happens with existing Google user agents and the token works only as a control. The same page states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.
Other AI companies run their own crawlers with their own user agent tokens; an llms.txt file neither allows nor blocks any of them.
llms.txt vs robots.txt vs sitemap
| File | Purpose | Who reads it |
|---|---|---|
robots.txt | Says which URLs automated clients may fetch | Search engine and AI crawlers, per their own policies |
sitemap.xml | Lists all indexable URLs with metadata | Search engines |
llms.txt | Curated, annotated map of the most useful pages, in Markdown | AI agents that choose to look for it |
The specification frames it the same way: robots.txt sets what access is acceptable, sitemaps list all pages for search engines, and llms.txt offers a curated overview used on demand when an agent needs information about a topic. A sitemap is not a substitute because it cannot include external URLs and is far too large for a context window.
When it is worth adding
- Documentation sites and developer tools, where coding agents follow links to API references.
- Businesses whose policies (delivery, returns, pricing, opening hours) get asked about in chat assistants.
- Any site where you want control over which pages an agent reads first.
Skip it if you cannot keep it current: a file pointing at removed pages is worse than none. Keep the list short, use absolute URLs, and review it after a migration like you review the robots.txt file.
Common mistakes
| Mistake | Effect |
|---|---|
| Serving the file as HTML with a template | Agents receive navigation and markup instead of the Markdown index |
| Listing every URL on the site | Defeats the purpose; the file no longer fits in context |
| Links to pages that are noindex or blocked for crawlers | Agents that respect robots.txt cannot fetch them |
| Expecting a ranking effect in Google | None is documented |
| Treating it as an opt-out from AI training | Only robots.txt rules do that |
How to check it
Fetch the file and look at the status and content type:
curl -sI https://example.com/llms.txt
Expect 200 and a text content type. Then read it: curl -s https://example.com/llms.txt | head -20 should start with a single # heading, not with <!DOCTYPE html>. Verify every link in the file returns 200, for example with the redirect checker, which tests up to 10 URLs at once. Finally, confirm that the pages you list are not blocked by a directive in the meta robots tag or by robots.txt; the robots.txt checker shows the exact rule that applies to a URL.
Why monitor it
IndexRise does not monitor llms.txt yet. The nearest checks are the ones that keep the links inside it valid: nightly status codes for the pages you list, robots.txt changes that would block them, and noindex appearing on them. If the pages in your llms.txt are in your IndexRise project, a broken link in the file shows up as a 404 or a lost-indexability event on the page itself the next morning.
Frequently asked questions
Does Google use llms.txt?
Google has not announced support for llms.txt in Search or in its crawler documentation. The file does not change how Googlebot crawls or indexes a page. Treat it as a convenience for AI agents, not as an SEO signal.
Is llms.txt the same as robots.txt?
No. robots.txt tells automated clients what they may fetch; llms.txt tells an agent where the useful content is. The proposal describes them as complementary, with different purposes.
Where does the file go?
At the site root as /llms.txt, or at any subpath such as /docs/llms.txt, in which case it covers the pages under that path. The file must be plain text in Markdown, not HTML.
What is llms-full.txt?
A convention some sites use for a single file holding the full text of their documentation, next to the shorter llms.txt index. It is not defined in the llms.txt specification text itself.
Will llms.txt stop AI companies from training on my content?
No. It only points agents to content. To restrict crawling you need robots.txt rules for the relevant user agents, such as Google-Extended for Google's Gemini training and grounding.
Do I need the .md copies of my pages?
The proposal recommends them so agents get clean text, but the llms.txt file works on its own as an index. Start with the file; add Markdown versions if your platform can generate them.
Updated: 2026-10-20