The SEO industry spent 18 months telling businesses to publish llms.txt files for AI visibility. We reviewed the server-log data, Google's own statements and the original specification. The file is not being read.
Home › Research › llms.txt and AI search
Whether llms.txt, widely recommended as an AI search visibility tactic since late 2024, is actually consumed by the AI surfaces that matter: Google AI Overviews and AI Mode, ChatGPT, Perplexity, Claude and Copilot. Our own audit methodology scored llms.txt presence as a positive signal, so we had a commercial incentive to find that it works. We found the opposite, and changed our methodology accordingly.
We triangulated three independent primary sources with non-aligned incentives: the Ahrefs server-log study of 137,210 domains (June 2026), Google's on-record statements from Search Relations plus its published AI optimisation documentation, and the llms.txt specification itself as scoped by its author, Jeremy Howard of Answer.AI. We required a minimum of three independent primary sources before letting any claim drive a recommendation, and we cross-checked adoption figures against a second dataset of roughly 300,000 domains scanned by SE Ranking.
No major AI search surface reads llms.txt in mid-2026. The file's only evidenced use case is helping AI coding agents parse developer documentation, a scenario that is close to irrelevant for a typical UK small business. As a result we removed llms.txt from our own scored audit signals: it is now reported as a neutral, non-penalising note rather than an AI visibility win.
The largest empirical dataset on llms.txt consumption comes from Ahrefs' analysis of 137,210 domains with live traffic, published 15 June 2026. The numbers are stark. 28% of domains in the sample had published an llms.txt file, a figure that skews high because the sample is SEO-aware. Of those files, 97% received zero requests of any kind in May 2026. Not from AI bots. Not from anything.
The 3% of files that were fetched paint an even less flattering picture. Only 19.5% of those requests came from named AI bots at all, and that figure is a ceiling, because a fetch is not evidence the contents were used. AI retrieval bots specifically, the ones that feed answers to users (OAI-SearchBot, PerplexityBot, Claude search), accounted for just 1.1% of requests. Slackbot's link-preview fetcher requested llms.txt files more often than PerplexityBot did.
Zero AI-bot requests went looking for llms.txt files that did not exist. If AI systems wanted this file, we would see them probing for it the way they probe for robots.txt. They are not looking.
Google has taken the unusual step of addressing llms.txt directly and repeatedly. Gary Illyes of Google Search Relations said in mid-2025 that Google does not support llms.txt and is not planning to. John Mueller compared the concept to the keywords meta tag, the canonical example of a signal search engines learned to ignore, and in June 2026 called the file "not done for search" and a "temporary crutch, perhaps to save some tokens" for AI coding tools parsing developer documentation.
Most decisively, Google's own "Optimizing for generative AI features" guide, published late May 2026, contains a mythbusting section stating plainly that you do not need special machine-readable files like llms.txt to appear in generative AI search. Google speaks only for Google, but Google's surfaces are the largest share of UK AI search exposure, and on its own surfaces this is ground truth.
The llms.txt specification was published by Jeremy Howard of Answer.AI on 3 September 2024. Read the spec itself and the scope is explicit: it exists to help LLMs and agents orient at inference time, for example pulling a coding library's documentation into context. It is explicitly "as opposed to for training," and it "controls nothing and blocks nothing."
The "AI visibility" framing was attached later by the SEO industry, on the speculation that AI platforms would one day reward the file. No platform ever said they would. That speculation then hardened into checklists, plugins and paid deliverables, which is how a developer-documentation convenience file ended up being sold to plumbers and photographers as search optimisation.
In fairness to the file's defenders, Google took both sides of this question within a single week in late May 2026. Days after Google Search published its "you do not need llms.txt" guidance, the Chrome team shipped an experimental llms.txt check inside Lighthouse's Agentic Browsing audits, noting that without the file, agents may spend more time crawling a site to understand its structure.
That is the seed of the only credible future case for llms.txt: an agentic-web play, where autonomous browsing agents use it as a site orientation map. It is worth watching. But in the Ahrefs data those agentic fetches amounted to roughly 1 in 1,000 requests. It is a signal of possible direction, not of present value, and present value is what a small business is paying for when it buys an "AI visibility fix."
Our free AI Visibility Snapshot checks your site against the signals with evidence behind them, not the ones with hype. One clear page, within 24 hours.
Get My Free Snapshot →Part of the AI Visible Research: Our Findings series. We publish the evidence behind our audit methodology, including the findings that forced us to change it. Questions or challenges to the method are welcome via our enquiry form.