Every SEO newsletter this year has told you to add one small file to your site. The largest search engine on earth has told you, in writing, that it ignores that file. Both things are true, and the useful answer is in the gap between them.
llms.txt is a proposed convention, not a standard, and nobody is obliged to read it. Google confirmed in June 2026 that Search ignores the file entirely. Large-scale crawler logs show the major AI bots almost never request it, and after two years it sits on roughly one site in ten. It does earn its place in one narrow case: technical documentation that AI coding assistants fetch live. For everyone else it is an hour of cheap insurance, not a ranking tactic, and it will not rescue a site whose actual pages are a mess.
What llms.txt Actually Is
The idea came from Jeremy Howard, co-founder of Answer.AI and fast.ai, who floated it in September 2024. It was not a marketing play. He was solving a real engineering problem.
A language model reading your website has a budget. Feed it a modern page and most of that budget disappears into navigation, cookie banners, sidebars, footers, and whatever your theme injects before the first sentence of actual content. The proposal was a plain markdown file at the root of your domain listing your important pages with a line of context each, so a model could find the good parts without eating the whole site to get there.
That is genuinely sensible. The specification has since been revised, with a second version landing in 2026, and the format itself is about as simple as web standards get: a heading, a short description, then linked sections.
The trouble was never the idea. It was what the industry did with it next.
What the Data Actually Says
Two years of enthusiastic advocacy have produced a fairly clear picture, and it is not the one in the newsletters.
Adoption is thin. Rankability found the file on 8.7% of the world’s top 1,000 websites as of June 2026. A wider SE Ranking study across 300,000 domains put it at 10.13%. After eighteen months of being described as essential, roughly nine sites in ten have not bothered.
Google is unambiguous. Guidance updated on 15 June 2026 states plainly that you do not need to create new files of any kind and that Search ignores llms.txt. No ranking boost, no penalty, no effect on AI Overviews. John Mueller and Martin Splitt spent an entire episode of Search Off the Record taking the premise apart, and their objection is more interesting than a flat refusal: HTML is already the standard, markdown offers no retrieval advantage, and stripping a page down to markdown throws away the link structure and heading hierarchy that help a model understand how your content fits together.
The crawlers mostly are not asking. An aggregate analysis of 515 million bot events found requests for the file to be a negligible share of AI-crawler traffic. GPTBot, ClaudeBot, PerplexityBot and the rest overwhelmingly skip it and parse your HTML like everyone else.
A convention only works when the other side agrees to honor it. robots.txt earned its authority over thirty years because crawler operators chose to respect it. This file has been asking politely for two.
If a model cannot make sense of your HTML, a markdown index will not save you. Semantic headings, real text instead of text baked into images, descriptive links, and a page that says what it is in the first paragraph will do more for machine readability than any file at your domain root. Fix the pages first. The file is the garnish.
Where It Genuinely Earns Its Place
There is one context where this file is not theater, and it gets buried under the SEO noise.
Technical documentation.
When a coding assistant needs to know how a library works, it fetches the documentation live, mid-task, with a limited context budget and no patience for a marketing site. A curated index pointing straight at the API reference saves it from crawling forty pages of tutorials to find one method signature. That is exactly the problem the original proposal described, and it is why Anthropic, Cursor and Vercel all publish one.
The one concrete positive result that circulates comes from the same world: Mintlify, a documentation platform, reported 436 visits from AI crawlers after adding the file, most of that traffic from ChatGPT. Notice the shape of that number. It is documentation, read by developer tools, in the exact scenario the format was designed for.
So the honest version is this. If you ship developer docs, an API, or a technical reference that AI coding tools consume, add it. It does real work. If you run a blog, a store, or a local business site, you are not in that scenario, and the SEO advice quietly borrowed a documentation solution and sold it to everybody.
The Case For Doing It Anyway
Here is the argument that survives all of the above, and it has nothing to do with rankings.
On 3 June 2026, Cloudflare confirmed that automated requests had passed human ones for the first time, 57.5% of HTML traffic against 42.5% from people. Their chief executive noted the crossover arrived eighteen months earlier than he had predicted. HUMAN Security put year-over-year growth in genuinely agentic traffic, meaning bots that act rather than merely read, at 7,851%.
The audience for your website is changing composition faster than almost anyone planned for. Against that backdrop, the file costs an hour to write and nothing to maintain. It cannot hurt you, because the search engine that ignores it ignores it neutrally. If adoption tips in two years, you are already there.
That is a real argument. It is just a very different argument from “add this and rank in ChatGPT,” and you should not let anyone sell you the second one wearing the first one’s clothes.
The Decision in One Table
| If you run | Do this | Because |
|---|---|---|
| Developer docs or an API reference | Add it, and keep it current | Coding assistants fetch docs live and the index genuinely saves them work |
| A blog or content site | Add it once, then forget it | An hour of cheap insurance, but expect nothing measurable from it |
| A store or booking site | Skip it, fix structured data | Schema markup is read by everything today and actually drives what agents can do |
| A site with messy HTML | Neither, yet | The file points at pages. If the pages are the problem, it points at the problem |
What We Got Wrong
Honest pass, as always.
We treated repetition as evidence. This file arrived in our reading through sheer volume, mentioned in enough places that it started to feel settled. Nobody in that chain was citing crawler logs. They were citing each other. Frequency is not verification, and we should have gone looking for the request data far earlier than we did.
We reached for a file instead of the pages. Adding something at the root of a domain feels productive in a way that rewriting a thin page does not. It is a checkbox with a filename. The uncomfortable truth is that the work which actually makes a site legible to a model is the same slow work that made it legible to a reader, and there is no artifact you can upload to skip it.
We confused being readable with being recommended. These are separate problems. A model can parse your site perfectly and still have no reason on earth to mention you. Machine readability is table stakes. Being worth citing is the actual game, and no file at your domain root has ever won it.
Add It, Then Go Do the Real Work
Write the file. It takes an hour, it costs nothing, and the traffic mix on the web is genuinely shifting under everyone’s feet.
Then close the editor and go fix the page you have been avoiding, the one that is three paragraphs of preamble before it says what it does. That page is costing you far more with both machines and people than a missing markdown index ever will.
The pattern holds here the way it holds everywhere else. The impressive-sounding fix was optional. The boring one was the job.
- It is a proposal, not a standard. No crawler owes it a response, and most large-scale log studies show they do not give one.
- Google ignores it, on the record. Guidance from June 2026 confirms no effect on Search or AI Overviews, in either direction.
- Adoption stalled near one site in ten. 8.7% of the top 1,000 sites, about 10% across a 300,000-domain sample.
- Documentation is the real use case. Coding assistants pull docs live, and a curated index saves them genuine work.
- Bots now outnumber people. Automated requests hit 57.5% of HTML traffic in June 2026, which is the strongest reason to bother.
- Readable is not the same as recommended. No root file makes your content worth citing.
⊕ Read more AI inspired posts on our AI Blog
⊕ Check out the latest AI News
About the Author: Gavin Stacey
