llms.txt: What It Is, How to Create One, and What It Does for AI Search

TL;DR: llms.txt is a proposal, published by Jeremy Howard in September 2024, for a Markdown file at yoursite.com/llms.txt that tells large language models which of your content matters and how to read it. robots.txt controls what crawlers can access; sitemap.xml lists every page; llms.txt is the short, curated version you actually want an AI to read. Only one line is required: an H1 with your site's name.

What is llms.txt?

llms.txt is a single Markdown file you put at the root of your domain. Its job is narrow: give a language model a clean map of your site so it can use your content at inference time, when it's answering a question, rather than weeks earlier when a crawler happened to pass through.

Jeremy Howard (co-founder of Answer.AI and fast.ai) proposed it and published the format at llmstxt.org on September 3, 2024. It's a community proposal, not something any AI vendor shipped or any standards body approved.

Why llms.txt matters for AI search

When an AI tool reaches for your site to answer a question, it has a small context window and very little time. It can't read your whole site. llms.txt hands it a short list of what to look at, in Markdown that models parse easily, instead of making it guess from your rendered HTML.

That's part of broader AI search optimization, the same goal as Generative Engine Optimization and Answer Engine Optimization: make your content easy for an AI to read and credit. llms.txt is one small piece of that.

How is llms.txt different from robots.txt and sitemap.xml?

All three files live at your domain root, which is why people mix them up. They do different jobs:

File	Audience	Job
robots.txt	Crawlers	Controls which URLs may be accessed
sitemap.xml	Search engines	Lists every indexable page (often thousands)
llms.txt	Language models	A short, curated guide to your most useful content

A sitemap is exhaustive; it's built for search engines and tries to list every page you have. llms.txt is selective, and meant to be short. You can have all three, and usually should.

One practical link between them: llms.txt only helps if the AI's crawler is allowed in. If your robots.txt blocks a retrieval bot like OpenAI's OAI-SearchBot, no curated index will bring it back, so make sure the engines you care about can actually reach you first. That's the first step in getting cited in ChatGPT.

What does an llms.txt file contain?

The format is deliberately minimal. In order, a file can contain:

• An H1 with the name of the site or project. This is the only required element.
• An optional blockquote (>) with a one- or two-sentence summary.
• Optional Markdown sections with any extra detail.
• Optional H2 sections containing Markdown link lists, where each item is [name](url) followed by an optional : and a short note.
• An optional section literally named ## Optional, which holds secondary links a model can skip when it needs a shorter context.

Each link points to a page, ideally one with a plain-text Markdown version (more on that below).

A minimal llms.txt example

# Your Product

> Your Product helps teams do X without Y. Open source, self-hostable.

## Docs

- [Quickstart](/docs/quickstart): get running in five minutes
- [API reference](/docs/api): endpoints, auth, and rate limits

## Optional

- [Changelog](/changelog): release notes

That's a complete, valid llms.txt. The H1 carries the name, the blockquote gives the one-line pitch, and the link lists point at the pages you want a model to cite.

Do you need llms-full.txt and Markdown page versions?

llms.txt is the index. To give models the actual content behind it, there are two companions you'll see in the wild:

• Markdown versions of your pages: the same URL with .md appended (for example /docs/api.md), so a model gets plain text instead of rendered HTML. This .md page-version convention is part of the original proposal.
• llms-full.txt: one expanded file that inlines more of your content, for when a model wants everything without following links. This one isn't in the formal proposal; it's an ecosystem convention that grew up around docs platforms like Mintlify.

You don't need either to start. A solid llms.txt index is the part worth doing first.

How do you create an llms.txt file?

You can write one by hand in a few minutes:

1. Add an H1 with your site or product name.
2. Add a one-line blockquote summary under it.
3. Add an H2 section (for example ## Docs or ## Product) and list your most important pages as [name](url): short note.
4. Save the file as llms.txt and upload it to your domain root.
5. Confirm it loads at yoursite.com/llms.txt.

Keep the list short. The whole point is curation: link the handful of pages you'd want an AI to cite, not your entire site. If a page has a cleaner plain-text version, point at that. And if you can, publish .md versions of those key pages so a model gets readable text instead of markup.

Prefer not to write it by hand? Generate your llms.txt free: give it your site and it drafts a valid file in seconds, no account needed.

On a hosted CMS where you can't drop a file at the domain root over FTP, your CMS's SEO plugin can handle it too (AIOSEO, for example, has an llms.txt toggle). Whichever route you take, check the result against the format rules and confirm the links resolve; a validator like llmstxtchecker.net flags a malformed file.

Common mistakes with llms.txt

A few things quietly defeat the purpose:

• Dumping every URL. That's what a sitemap is for. If your llms.txt is exhaustive, it stops being a shortlist and stops being useful.
• Pointing at pages that read badly as plain text. If a linked page is mostly JavaScript or interactive widgets, a model gets little from it. Link the pages that hold up as plain text.
• Writing it once and forgetting it. When your important pages change, the file goes stale. Treat it like your sitemap and update it when the site does.
• Expecting it to carry weak content. llms.txt helps an engine find your best pages; it doesn't make thin pages worth citing. The content still has to earn the mention.
• Assuming that publishing it means you're read. In practice, most published files just sit there unread. If there's a specific AI tool you want using yours, you often have to point it at the file directly, and keep it current, so it's worth reading when something finally does.

Who's adopting llms.txt?

Adoption shows up first in documentation tooling, where clean link lists are a natural fit. The directory at llmstxt.site tracks public implementations, and several docs platforms (among them VitePress, Docusaurus, Drupal, and nbdev) have ways to generate or serve one.

Which AI engines actually read llms.txt?

Fewer than the hype suggests. The clearest signal comes from an Ahrefs study of 137,210 domains in mid-2026: 28% had published an llms.txt file, but 97% of those files were fetched by nothing at all that month. Of the requests that did arrive, most came from SEO audit tools and general crawlers; the AI retrieval bots that actually power answers (PerplexityBot, OpenAI's OAI-SearchBot) made up just 1.1% of AI-bot requests.

Google has been openly skeptical, too. Its John Mueller compared llms.txt to the old keywords meta tag and argued that an LLM "can't trust what is here as a way of differentiating between different websites." So don't assume that publishing the file gets you read by Google's AI Overviews or any other big engine today.

Where llms.txt clearly earns its place is documentation and coding tools: assistants like the ones in your code editor that pull a library's docs on demand. That's a real, working use case, and it's exactly where adoption started. If you maintain a developer-facing product, that's the case to optimize for: a clean llms.txt, plus .md versions of your reference pages, lets a coding assistant answer "how do I use this API?" straight from your docs instead of from a guess or a stale training snapshot.

How do I know if my llms.txt is being read?

Check your server access logs for the AI retrieval bots, OAI-SearchBot and PerplexityBot, and see whether either has fetched the file. You can also test it yourself: paste your llms.txt URL into a coding assistant and ask it something that's only answerable from a linked page, then watch whether it actually pulls the file. Or skip the log-digging: a citeproduct scan checks whether AI bots can reach your site and whether your llms.txt is present and readable.

Does llms.txt move the needle?

The file alone won't get you cited. It's still worth the ten minutes it takes to write one, since it does real work in the places that read it. For a typical marketing site, treat it as optional: a small, cheap bet on where AI search is heading rather than something that lifts your rankings today.

The bigger question is whether AI engines can understand and trust your site at all. llms.txt is one input; your structure, your schema, and whether models actually name you matter just as much. citeproduct measures all of that: it scores how readable and citable your site is, and checks whether AI engines mention you. Scan your site's AI readiness to see whether engines can read and cite you today, or read how we score it.