What Is llms.txt? What the File Does and Doesn't Do, Measured in 27 Days of Server Logs (2026)

llms.txt is a Markdown text file that you place at your-domain.com/llms.txt. Its job is to tell language models and AI agents, in compact form, what your website is about and to link to your most important content. The idea comes from Australian data scientist Jeremy Howard (Answer.AI), who proposed it in September 2024. A revised version 2 of the specification was published in August 2026.

Few files generate as much noise right now. SEO tools check for it, plugins create it with one click, agencies sell it as a must-have for “AI visibility”. We wanted to know what is actually behind it, so instead of reading opinions we measured. We analysed 27 days of server logs from getmind.io (25 September to 22 October 2026, 218,657 requests), checked 73 well-known domains for an llms.txt, and rebuilt the Lighthouse audit with six purpose-built test servers.

Three results up front:

1. Not a single major AI crawler requested /llms.txt in 27 days. Not GPTBot, ClaudeBot, PerplexityBot, Googlebot or bingbot. The same bots fetched our robots.txt hundreds of times over the same period.
2. Lighthouse does not penalise you for not having an llms.txt. A missing file (404) counts as “not applicable”. The audit only fails if you serve a broken file. Many articles, our own included, have described this incorrectly.
3. A different route actually did reach us: 2,386 requests from AI search services asked for our pages directly as Markdown, using the HTTP header Accept: text/markdown. That is a signal worth taking seriously.

In this article you will learn how an llms.txt is structured, who really reads it, what Google says about it, how to create one if you want it anyway, and which websites it is worth the effort for.

In this video from June 2026, Christopher Penn and John Wall of Trust Insights discuss how to get started with llms.txt and which kinds of content the file makes sense for.

What is llms.txt? The short version

Websites are built for people. A typical article page consists of navigation, a cookie banner, a sidebar, scripts, tracking and a footer, and somewhere in between sits the actual text. We counted this on three of our own articles. For our article What is IndexNow? the server delivers 78,815 bytes of HTML, of which 25,165 bytes are readable text, roughly a third. The sitemap guide is similar.

A human simply ignores the rest. A language model has to filter it out first, and every unnecessary character costs context window, time and money. The idea behind llms.txt is therefore simple:

• One file at a fixed address (/llms.txt) that an agent can find without guessing, much like robots.txt.
• A short overview in Markdown: what kind of website this is, what lives where, what matters.
• Links to clean versions of the content, ideally also as Markdown files, so the agent does not have to dig through HTML.

It is just as important to understand what llms.txt is not. It is not an access rule. Whether a crawler may read your pages is still controlled by robots.txt. It is not a sitemap either. The XML sitemap lists all URLs in machine-readable form for search engines; llms.txt is a curated, annotated selection for language models. And it is not a ranking factor. No search engine has ever confirmed that the file improves visibility.

How is an llms.txt file structured?

The specification on llmstxt.org is short. A valid llms.txt consists of Markdown sections in a fixed order:

1. An H1 heading with the name of the site or project. This is the only required part.
2. A blockquote (>) with a short summary.
3. Any number of paragraphs or lists without headings containing further notes.
4. Any number of H2 sections, each containing a list of links. Each entry has the form [name](URL), optionally followed by a colon and a description.

A minimal example:

# Example Ltd

> Example Ltd sells spare parts for coffee machines
> and ships within the UK in 1 to 2 working days.

Prices include VAT. Returns are free within 30 days.

## Products

- [Gaskets](https://example.com/gaskets.md): Brew group gaskets for all common brands
- [Descaler](https://example.com/descaler.md): Liquid and tablets

## Service

- [Shipping and delivery times](https://example.com/shipping.md)
- [Returns](https://example.com/returns.md)

## Optional

- [Company history](https://example.com/about.md)

The ## Optional section is a convention for secondary material that an agent may skip when context is tight. In version 1 it had a fixed technical meaning for tools that expand an llms.txt into one big context block. In version 2 that special meaning was dropped. The section is still allowed, but it is only a hint now.

What version 2 (August 2026) changed

Most guides on the web still describe version 1. The August 2026 revision brings three changes that matter in practice:

• Subpaths are officially defined. An llms.txt does not have to sit at the root. A file at /docs/llms.txt covers every page under /docs/. If there are several, the agent uses the most specific one. Google does exactly this: ai.google.dev/llms.txt returns 404, but ai.google.dev/gemini-api/docs/llms.txt exists (37,731 bytes, measured 22 October 2026).
• Markdown versions may use two URL forms. Either append .md to the full URL (page.html.md) or replace the extension (page.md).
• There is an official way to discover the files. Pages can point to their Markdown version via <link rel="alternate" type="text/markdown"> and to the responsible llms.txt via <link rel="describedby">. This also works as an HTTP Link: header, without touching the pages themselves.

Version 2 also describes, for the first time, how the file is meant to be read: the agent looks at the llms.txt, picks what it needs and follows only the relevant links. The file should stay small; the detail lives behind the links.

And what is llms-full.txt?

llms-full.txt is a convention introduced mainly by documentation platforms such as Mintlify. Instead of a link list it contains the full content of every page in a single file. It is not part of the official specification.

How large these files get surprised us. Measured on 22 October 2026:

69 megabytes of text is roughly 17 million tokens. No language model can read that in one go. A file like that is at best a download for local search, not context material. That is exactly why version 2 stresses that the file should stay small and link to details.

We ran the same numbers for getmind.io. An llms.txt with all 93 English articles (title, link, description) would be 32,167 bytes, about 8,000 tokens, which fits easily into any context window. An llms-full.txt with the full text of every article would come to 2.6 MB or about 650,000 tokens, and would fail on size alone with almost every model.

Who actually reads llms.txt? Our log analysis

This is the decisive question, and it cannot be answered with opinions, only with log files. getmind.io has no llms.txt. Anyone requesting it gets a 404. That is precisely what makes the measurement clean: every request in the log comes from a visitor who went looking for the file on their own.

We merged all Caddy access logs from 25 September to 22 October 2026: 218,657 requests to getmind.io. They contain 78 requests for /llms.txt, plus 2 for /.well-known/llms.txt and one for /llms-full.txt. That is just under three a day.

Now the senders:

More than two thirds of the requests came from a single SEO audit tool. The rest are other SEO and analytics tools plus requests with ordinary browser user agents, which typically come from scanners or from people checking manually. Not a single request came from a crawler run by an AI provider or a search engine.

You can rule out that these bots simply rarely visit. Over the same period they crawled our site heavily and, as a matter of course, requested robots.txt:

An honest caveat: these figures are based on the user agent, which can be faked. We saw this in the logs. The IP address 34.153.192.236, from a cloud data centre, appears as “GPTBot”, “ClaudeBot” and “bingbot” at the same time, so it is certainly none of them. The absolute request counts are therefore too high. For the core finding it makes no difference. If a real AI crawler had looked for llms.txt, the request would be in the log, under whatever name. There is none.

This matches what other site owners report and what the providers say. Neither OpenAI, Anthropic, Perplexity nor Google has confirmed that their crawlers process llms.txt. If our analysis of AI crawlers taught us anything, it is this: crawlers fetch robots.txt and HTML pages. What marketing slides say about their preferences rarely shows up in the logs.

Where llms.txt does get read

That does not make llms.txt useless. The file does get read, just not by crawlers harvesting the web for training data or search indexes. It is read by agents carrying out a specific task, mostly when a human sends them there:

• Coding agents such as Claude Code, Cursor or GitHub Copilot, when a developer writes: “Read the docs at docs.example.com/llms.txt and build me the integration.”
• MCP servers and documentation tools that load llms.txt files on purpose to give a model a library’s documentation.
• People who paste the URL into a chat window because they know the content is cleanly prepared there.

That explains why adoption is so uneven. In developer documentation llms.txt has long been standard, because that is where agents with concrete tasks are at work. On a blog, a company website or a shop this use case rarely comes up.

Adoption: we checked 73 domains

On 22 October 2026 we used curl to check which well-known websites serve an llms.txt. The selection is deliberately mixed: developer docs, hosting providers, shop systems, SEO tools, large German shops and news sites. We only counted responses with status 200 and a text or Markdown content type.

Have a real llms.txt (31 of 73): docs.anthropic.com, developers.openai.com, docs.github.com, github.com, docs.docker.com, docs.stripe.com, stripe.com, cloudflare.com, developers.cloudflare.com, vercel.com, netlify.com, render.com, fly.io, railway.com, supabase.com, nextjs.org, react.dev, vuejs.org, nuxt.com, svelte.dev, nodejs.org, bun.sh, mintlify.com, zapier.com, docs.hetzner.com, wordpress.org, shopify.com, woocommerce.com, yoast.com, semrush.com and otto.de.

No llms.txt (404 or blocked): among others anthropic.com, openai.com (403), ai.google.dev (subpath only), huggingface.co, kubernetes.io, docs.python.org, laravel.com, symfony.com, tailwindcss.com, astro.build, nginx.org, caddyserver.com, hetzner.com, digitalocean.com, ahrefs.com, moz.com, sistrix.de, wikipedia.org, ebay.de, amazon.de, zalando.de and every German news site we tried (heise.de, golem.de, t3n.de, chip.de, spiegel.de, zeit.de, faz.net, tagesschau.de, bild.de).

The pattern is clear. Almost all hits are developer documentation and platforms whose customers are developers. None of the German news sites has an llms.txt; among the large German shops only otto.de has one. The Anthropic difference is telling too: the docs at docs.anthropic.com have one, the company site anthropic.com does not. That is exactly the line Google draws as well: useful for documentation, not for most other websites.

Two findings from the sample are instructive:

• WooCommerce serves a file generated automatically by Yoast SEO. Its first line is not an H1 but the note “Generated by Yoast SEO v26.0, this is an llms.txt file…”, with the heading only after that. Strictly speaking that breaks the order required by the spec. Lighthouse still finds the H1 because it searches the whole file.
• deno.com, netcup.de and sueddeutsche.de answer /llms.txt with status 200 but deliver a full HTML page, over 2 MB in the case of sueddeutsche.de. That is a so-called soft 404. Harmless for people, but a failure for Lighthouse. The next section shows why.

What does Google say about llms.txt?

Google makes two statements, and they only contradict each other at first glance.

Google Search is clear. In its guide “Optimizing your website for generative AI features on Google Search” from 15 May 2026, Google says, in essence, that you do not need to create new machine-readable files, AI text files, special markup or Markdown to appear in Google Search, including AI Overviews and AI Mode, because Google Search does not use them. John Mueller had earlier compared llms.txt to the keywords meta tag, a statement site owners make about themselves that a search engine therefore cannot take at face value. When Lily Ray asked him shortly afterwards why Google still offers llms.txt files on its own properties, he stressed that the two things are unrelated: developer documentation has different readers than Google Search.

Chrome Lighthouse has, since version 13.3.0 (7 May 2026), checked by default in its new “Agentic Browsing” category whether an llms.txt follows the recommendations. That is the part quoted everywhere as a contradiction. We read the code and tested the behaviour ourselves.

We rebuilt the Lighthouse audit

In the Lighthouse 13.4.1 source (core/audits/agentic/llms-txt.js) the logic is short. Lighthouse fetches /llms.txt and checks three things: is there a line starting with # (H1)? Is there at least one Markdown link [...](...)? Is the file at least 50 characters long? More interesting is what happens with the status code before that.

We wrote a small Python test server that answers /llms.txt in six different ways and ran Lighthouse 13.4.1 against each variant:

This leads to something stated incorrectly in many articles, including our own article on Google and AI SEO, which we corrected after this test: Lighthouse does not flag sites that have no llms.txt. Chrome’s own documentation says so explicitly: if the file is missing, the audit is “Not Applicable” because it is “optional at the moment”. We ran getmind.io through the “Agentic Browsing” category ourselves. Result: full score, with the llms.txt audit marked “not applicable”.

It turns red in two cases. Either your server returns a 5xx error for /llms.txt, which points to misconfigured rewrite rules. Or you serve a file that does not follow the recommendations. That can be a careless plugin output or a soft 404 like the ones at netcup.de and sueddeutsche.de. Paradoxically, a website with no llms.txt at all does better in Lighthouse than one with a broken file.

The weighting is reassuring too. In the “Agentic Browsing” category llms.txt is just one of six equally weighted audits. It has no influence on the Performance, SEO or Accessibility scores.

The route that actually reached us: Markdown via content negotiation

While searching the logs for llms we came across something we had not been looking for. 2,386 requests in the period contained text/markdown in their Accept header. The client is telling the server: “If you have this page as Markdown, please give me that. Otherwise I’ll take HTML.”

That is a completely different pattern from llms.txt. These bots are not looking for an extra file in the hope that it exists. They ask for a leaner version of every page they are fetching anyway. Our server still sends them HTML because we do not offer Markdown versions. But the demand is measurably there, and it comes from services that prepare content for AI answers.

Version 2 of the llms.txt specification picks up exactly this idea. Rather than relying only on one central file, it recommends offering every page as .md as well and pointing to it with a Link header. If you are thinking about machine-readable content for AI, start here. Per-page Markdown versions are more useful in the long run than a single overview file. What a Markdown file is and how it is structured is covered in our article What is an .md file?.

How to create an llms.txt: step by step

If after all this you want an llms.txt, for example because you run documentation, an API or a software product, it takes a few minutes.

Step 1: Pick the content

An llms.txt is not a sitemap. Do not include every URL, only the 10 to 50 pages an agent is most likely to need to answer questions about your offering: getting started, key guides, API reference, pricing, contact, terms. Everything else goes into ## Optional or stays out.

Step 2: Write the file

Stick to the order: H1, blockquote, optional prose, then H2 sections with link lists. Each description after the colon should say in one sentence what the agent will find behind the link. That is the real added value over a sitemap.

# My Product

> My Product is an open-source library for XY.
> Current version: 3.2. Licence: MIT.

## Getting started

- [Installation](https://docs.example.com/install.md): npm, pnpm and Docker
- [Quickstart](https://docs.example.com/quickstart.md): First project in 5 minutes

## Reference

- [API](https://docs.example.com/api.md): All functions with parameters and examples
- [Configuration](https://docs.example.com/config.md): Every option in config.json

## Optional

- [Changelog](https://docs.example.com/changelog.md)

Step 3: Serve it correctly

Put the file at /llms.txt in your root directory; for static site generators like Astro or Next.js that means the public/ folder. Then check three things:

curl -sI https://your-domain.com/llms.txt | grep -iE "^HTTP|content-type"
# Expected: HTTP/2 200 and content-type: text/plain or text/markdown

curl -s https://your-domain.com/llms.txt | head -3
# Expected: first line starts with "# "

curl -s -o /dev/null -w "%{http_code}\n" https://your-domain.com/llms-txt-does-not-exist.txt
# Counter-check: must return 404, otherwise you have a soft 404

The counter-check matters. If your server answers arbitrary paths with status 200 and an HTML page, every tool will treat even a missing llms.txt as present and broken. That is exactly what happens with the three domains from our sample.

Step 4: Optionally check with Lighthouse

lighthouse https://your-domain.com/ --only-audits=llms-txt --output=json --quiet

The audit takes about 20 seconds. If it passes, the file is formally fine. Whether anyone reads it, only your server log can tell you.

Step 5: Measure requests

This is the step almost every guide leaves out. If you go to the trouble of creating the file, measure who fetches it. With Caddy and JSON logs a one-liner does it:

grep '"uri":"/llms.txt"' /var/log/caddy/access.log \
  | grep -o '"User-Agent":\["[^"]*' | sort | uniq -c | sort -rn

With nginx or Apache in the default format:

grep ' /llms.txt ' /var/log/nginx/access.log | awk -F'"' '{print $6}' \
  | sort | uniq -c | sort -rn

After four weeks you will know whether it was worth it. If only SEO tools show up, you have your answer.

Generate it automatically?

For WordPress, Yoast SEO and Rank Math create the file with one click, and documentation platforms like Mintlify do it automatically. That is convenient, but check the result. Auto-generated files often just list every post, without curation and without useful descriptions. That removes exactly what distinguishes an llms.txt from a sitemap.

Webflow’s explainer is from November 2025, slightly older than our usual six-month limit. It covers the basic idea well; version 2 of the spec and the Lighthouse check are not in it yet, both are covered above.

Risks and misconceptions

An llms.txt costs little, but not nothing. A few points are worth knowing before you publish one.

It is public and invites copying. A cleanly prepared summary of your most important content is convenient for scrapers too. If you want to protect your content from AI training, you do that via robots.txt and access controls, not by leaving out an llms.txt. Conversely, do not put anything in the file that is not also on your public pages.

It needs maintenance. An llms.txt pointing to renamed or deleted pages is worse than none. An agent follows the links and ends up nowhere. The best approach is to generate it automatically from your content at build time and curate only the selection by hand.

It does not control access. Some guides claim you can use llms.txt to define which content AI may use. That is not true. The specification has no allow or deny rules. That remains the job of robots.txt.

It is not a lever for rankings or citations. Anyone promising that an llms.txt will make your website more visible in ChatGPT, Perplexity or AI Overviews is promising something no provider confirms and that we could not trace in our logs. What gets cited in AI answers is decided by the content itself. More on that in our article on Google, AI Overviews and AI SEO.

Who is llms.txt worth it for?

Based on our measurements the answer is fairly clear.

Worth it:

• Developer documentation, APIs, SDKs and open-source projects. Coding agents genuinely use the file here, and developers paste the URL into their tools on purpose.
• Software and SaaS products whose customers build integrations with AI assistants.
• Websites that already have Markdown versions. Then the llms.txt is just a link list and done in ten minutes.

Fine to do, but measurably low impact:

• Shops, company websites, blogs and magazines. The effort is small, so it does no harm as long as the file is correct. Just do not expect any effect on search or AI citations.

Do something else first:

• If you do not yet have a clean robots.txt, an up-to-date sitemap and IndexNow, start there. These files and protocols are demonstrably read; we see it in our logs every day.

And getmind.io?

We have deliberately decided against an llms.txt, at least for now. Our readers are mostly people arriving from search, and in 27 days not a single AI crawler asked for the file. We find the 2,386 requests for Markdown more interesting. If we invest time in machine-readable content, it will be Markdown versions of our articles served via content negotiation. When we implement that, we will measure the effect and add the result here.

FAQ about llms.txt

What is llms.txt in simple terms?

A Markdown file at /llms.txt that tells language models and AI agents in a few lines what a website is about and links to its most important content. Think of it as an annotated table of contents for machines.

Is llms.txt an official standard?

No. It is a proposal by Jeremy Howard (Answer.AI) from September 2024, revised as version 2 in August 2026. There is no RFC and no standards body behind it, unlike robots.txt (RFC 9309).

Do ChatGPT, Claude or Perplexity read my llms.txt?

Their crawlers apparently do not. In our 27 days of logs, none of the crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft or Apple fetched the file, even though the same bots requested robots.txt hundreds of times. It gets read when a user deliberately points an agent at it, for example while coding against third-party documentation.

Does llms.txt help with Google and AI Overviews?

No. In its guide from 15 May 2026, Google states explicitly that Google Search does not use such files, including for AI Overviews and AI Mode.

Why does Lighthouse check for llms.txt then?

Lighthouse’s “Agentic Browsing” category checks how well AI agents can work on a page. That has nothing to do with search. Importantly, if the file is missing, the audit is “not applicable”. You can only fail with a faulty file or a server error.

What is the difference between llms.txt and robots.txt?

robots.txt controls which crawlers may access which areas and is read by every reputable crawler. llms.txt controls nothing; it offers a curated overview. It does not replace robots.txt and cannot block access.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short link list with descriptions. llms-full.txt contains the full content of every page in one file. It is not part of the specification and quickly becomes huge for large docs, 69 MB at developers.cloudflare.com, for example.

Where does the llms.txt file go?

At the root of your domain at /llms.txt. Since version 2, subpaths like /docs/llms.txt are also allowed; such a file then covers all pages under that path.

What content type should llms.txt use?

text/plain or text/markdown, both with UTF-8. Both are common in practice: docs.github.com and docs.docker.com serve text/markdown, react.dev and nodejs.org text/plain.

Can an llms.txt hurt?

Not directly. But a faulty file fails the Lighthouse audit, outdated links send agents nowhere, and a well-prepared summary makes scrapers’ work easier. Without a file you have none of these problems.

Do I need a plugin for WordPress?

Yoast SEO and Rank Math can generate the file. You can just as well write it by hand and upload it to the root directory. Always check a plugin’s output, because auto-generated files often just list every post.

How do I find out whether my llms.txt is being read?

Only through your server log. Filter requests for /llms.txt by user agent, as shown in the “Measure requests” section. Browser-based analytics tools cannot see these requests because no JavaScript runs.

Conclusion

llms.txt is a sensible idea with a clear but narrow use case. For developer documentation and APIs the file is now standard, and coding agents really do use it there. For every other kind of website it is, so far, mostly a checkbox in SEO tools. Our measurement is unambiguous: 78 requests in 27 days, more than two thirds of them from a single SEO audit tool, zero from the crawlers of AI providers and search engines.

If you create an llms.txt, do it properly: short, curated, with an H1 and links, no soft 404, and a look at your logs after four weeks. If you do not create one, by today’s evidence you lose nothing, not even in Lighthouse. And if you really want to do something for machine-readable content, check whether requests with Accept: text/markdown are already reaching you. For us it was 2,386.