What is llms.txt?
llms.txt is a plain-text file that lives at the root of your website —yourdomain.com/llms.txt — and gives AI systems a curated, markdown-formatted map of your site: what it is, what it offers, and which pages matter most. It was proposed by Jeremy Howard of Answer.AI in September 2024 as a way to help large language models make sense of websites without crawling and parsing every page of HTML.
Where a human visitor gets navigation, design, and context clues, an AI model fetching your site gets a wall of markup. llms.txt cuts through that: one small file that says "here is who we are, and here are the ten pages that actually explain us." Models with limited context windows can read it in a single pass.
llms.txt vs robots.txt vs sitemap.xml
These three files answer different questions and work together, not in competition:
- robots.txt — permission. Tells crawlers (including AI bots like GPTBot and ClaudeBot) which parts of your site they may access.
- sitemap.xml — inventory. Lists every indexable URL so search engines can find them all.
- llms.txt — understanding. A short, prioritized, human-readable guide that helps AI models grasp what your site is about and where the substance lives.
The format, in brief
An llms.txt file is standard markdown with a fixed shape: one H1 with your site or product name, a blockquote with a one-line summary, optional paragraphs of context, then H2 sections (Docs, Guides, API — whatever fits) containing bulleted links with short descriptions. A final section named Optional marks links AI models can skip when context is tight. The generator above produces exactly this structure.
Should you bother?
Honest answer: llms.txt is an emerging convention, not a ratified standard, and no AI vendor guarantees they read it. But adoption is real — documentation platforms generate it automatically and thousands of sites publish one — and the cost is a single text file. If you care about how ChatGPT, Perplexity, and Claude represent your product, it is among the cheapest AI-visibility bets you can make. Pair it with proper structured data and an AI-friendly robots.txt (our AI Readiness Auditchecks all three).
Common questions
- Is llms.txt an official standard?
- It is a community proposal, introduced by Jeremy Howard of Answer.AI in September 2024, not a ratified web standard. Adoption is growing: documentation platforms like Mintlify generate it automatically, and thousands of sites publish one. Since it costs a single text file, most teams treat it as cheap insurance.
- Does llms.txt replace robots.txt or sitemap.xml?
- No. robots.txt controls which crawlers may access your site, sitemap.xml lists every URL for search indexing, and llms.txt is a curated, human-readable map that helps AI models understand what your site is and which pages matter most. The three work together.
- Where do I put the llms.txt file?
- At the root of your domain, so it resolves at yourdomain.com/llms.txt — the same location pattern as robots.txt. On most static hosts, drop the file in your public folder and deploy.
- What goes inside an llms.txt file?
- A markdown document with one H1 title for your site, a blockquote summary, optional context paragraphs, and H2 sections containing lists of your most important links with short descriptions. An "Optional" section can mark links that AI models may skip when context is limited.