What Is llms.txt and Do You Actually Need It?

llms.txt is a proposed standard: a plain-text, Markdown-formatted file at yourdomain.com/llms.txt that gives AI models a curated list of your most important content. Here is the honest answer to “do I need it”: for most websites, not yet. No major AI system – Google, OpenAI, or Anthropic – has confirmed it uses llms.txt, so the file rarely changes anything today. It is cheap to add and harmless to have, but it is not the AI-visibility lever people hope it is. This guide explains what it does, who it actually helps, how to create one, and what really moves the needle.

The topic is noisy because the file sounds like a quick win. It is not a ranking button. Treat it as a low-cost experiment, not a strategy.

What is llms.txt?

llms.txt is a file you place at the root of your site (for example, yourdomain.com/llms.txt) written in Markdown – the same lightweight format used in GitHub README files. Instead of letting AI crawlers wander your entire site, it hands them a short, structured index: a title, a one-line description of your business, and grouped links to your key pages with a sentence explaining each. The idea, from the proposal at llmstxt.org, is that large language models have limited context and struggle to parse cluttered HTML, so a clean Markdown summary makes your important content easier to find and quote.

Sources and further reading

This explainer is grounded in the specification, platform statements, and recognized industry sources:

1. llmstxt.org – the official /llms.txt proposal and format specification.
2. Google Search Central – Optimizing your website for generative AI features on Google Search.
3. John Mueller (Google) – public statement that no AI system currently uses llms.txt.
4. Semrush – What Is LLMs.txt and Should You Use It? (server-log test and adoption data).
5. Ahrefs – What Is llms.txt, and Should You Care About It?
6. NerdyData – adoption tracking of published llms.txt files.

That is Google saying, in its own documentation, that it ignores llms.txt. Keep it in mind before you invest much time here.

What problem is llms.txt meant to solve?

The proposal targets two real frustrations. First, modern pages are hard for machines to read: much of the content loads through JavaScript, and the HTML is buried under navigation, ads, and scripts. Second, large sites bury their best material – an AI crawler cannot easily tell your flagship product page from a five-year-old blog post. A curated Markdown file is meant to cut through both problems by pointing models straight at the content you want them to use. It is a reasonable idea. The catch is whether the engines actually read it.

llms.txt vs robots.txt vs sitemap.xml

A lot of the confusion comes from treating llms.txt like robots.txt. They are not the same thing.

robots.txt controls access and sitemap.xml aids discovery, and both are honored. llms.txt tries to shape understanding, and that is exactly the part no engine has committed to.

Do the AI engines actually use it?

This is where the hype meets reality. As of now, none of the major AI companies – OpenAI, Google, or Anthropic – has said its crawler follows llms.txt, and Google’s John Mueller has publicly stated that no AI system currently uses it. Adoption is also thin: according to NerdyData, only around 951 domains had published an llms.txt file as of mid-2025, a tiny slice of the web.

There is even test evidence. In a published experiment, Semrush added llms.txt to Search Engine Land and then checked the server logs: over roughly two and a half months, the file received zero visits from the major AI crawlers – GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Traditional crawlers touched it only a handful of times. In other words, the bots you are trying to help are not reading the file.

Who is using llms.txt today?

Adoption is concentrated in one place: software and developer-documentation companies. Early adopters include the likes of Anthropic, Hugging Face, Vercel, Zapier, and Cal.com, and their files almost always index technical docs and API references rather than a marketing site. The structures vary widely – some use deep heading hierarchies with full code examples, others are just a flat list of links with short descriptions – which tells you the format is still settling. The takeaway is clear: llms.txt is a docs-and-developer convention right now, not a mainstream marketing tactic. If your site is not documentation-heavy, you are not missing out by waiting.

So do you actually need llms.txt?

For the large majority of sites, no – not as a priority. The realistic picture:

• Most businesses can skip it for now. It will not lift your rankings or your AI citations today, because the engines are not using it.
• Documentation-heavy and developer sites are the exception. If you publish large technical docs or an API reference, a clean llms.txt can help tools and agents that DO parse Markdown, and the early adopters are almost all in this camp.
• It is a cheap, low-risk experiment. Adding the file does no harm, so if you are curious and have a spare hour, there is no downside to trying it and watching your logs.
• Do not let it replace real work. The danger is spending a day on llms.txt while ignoring the things that genuinely drive AI visibility.

How to create an llms.txt file (if you want to)

If you decide to experiment, it takes three steps and usually a developer for the upload:

1. Choose the content to feature. List your most important pages – core product or service pages, current cornerstone posts, pricing, about, and contact – not your whole archive.
2. Write the file in Markdown. Start with an H1 for your site name and a one-line description, then group links under H2 headings with a short sentence describing each link. Save it as a plain text file named llms.txt.
3. Upload it to your root. Place it in your root directory so it resolves at yourdomain.com/llms.txt (or in the relevant subdirectory for a docs subdomain), then open the URL to confirm it loads. Keep it updated as your key pages change.

Common myths about llms.txt

Most of the hype comes down to a few misconceptions worth clearing up:

• “It boosts your AI rankings.” There is no ranking or citation effect today, because the engines are not reading the file.
• “It works like robots.txt.” robots.txt is an enforced standard; llms.txt is an unadopted suggestion.
• “Adding it gets ChatGPT to cite you.” Citations come from authority, clean structure, and third-party mentions – not from a text file.
• “Bigger is better.” A bloated, outdated file helps nobody. If you do publish one, keep it short and current.

What actually moves the needle for AI visibility

If your goal is to be recommended and cited by ChatGPT, Perplexity, and Google AI Overviews, spend your time on the work the engines demonstrably reward – the core of generative engine optimization: a clear, consistent brand entity; answer-first content the models can quote; valid structured data; and citations on the third-party sources AI answers trust.

Our AEO checklist walks through the exact steps, and our AI Visibility and GEO team runs the program end to end. For proof it works, see the AI visibility case study where AEO, GEO, and SEO turned into real leads – no llms.txt required.

The bottom line on llms.txt

llms.txt is a sensible idea waiting for adoption that has not arrived. Add it if you run heavy documentation or simply want to experiment, but do not expect it to change your AI visibility, and do not let it distract from the work that does. If you want to know where you actually stand in AI answers, request a free AEO and GEO audit.