The web's biggest sites mostly skip llms.txt. The data says that's fine.
We checked the 1,000 most popular domains on the web for an llms.txt file, graded every one we found, and tested whether having one gets a site into Google's AI answers. Here's who publishes one, what's in it, and whether it matters.
A table of contents for AI tools. Nothing more.
An llms.txt file is a plain text file at yourdomain.com/llms.txt. It lists a site's key pages with short descriptions so an AI tool can find its way around. It's a community proposal, not an official standard, and no search engine or AI company is required to read it.
Over the past year it's been sold as the way to "get into AI search." SEO plugins now create one automatically, and agencies charge to write one. So we asked three questions: who actually publishes one, what's in them, and does having one change anything?
It's a software-industry habit, not a web standard.
About one in five of the most popular websites we could reach publish an llms.txt: 127 of 620 (20.5%). Counting each company once instead of each domain, it's one in four. Nearly two-thirds of those files come from software, SaaS and developer-tool companies.
The adopters read like a developer's bookmarks bar: Cloudflare, GitHub, Stripe, Shopify, HubSpot, Slack, Dropbox and Sentry. Many are companies whose customers use AI coding tools that read their documentation.
Only 7 of the 64 reachable top-100 websites publish one (11%), against 22% further down the list. That gap shrinks and stops being statistically clear once you count each company once (Google, Amazon and Microsoft own many top-100 domains between them). Either way, the biggest names aren't leading.
In our separate study of 464 home services companies, 26% had an llms.txt, more than the web's biggest sites. About a third of those carried an SEO plugin's signature. Small-business sites often got theirs from a plugin update, not a decision.
search engines, portals and social platforms we could reach publish one. The companies building AI search and AI assistants mostly don't use llms.txt on their own sites.
More sites fake an answer than publish a real file.
When you ask a website for a page that doesn't exist, it should answer "404, not found." But 136 of the 620 websites (22%) answer a request for /llms.txt with a web page or other non-file response, usually their homepage, and a "200 OK" success code. That's more than the 127 that publish a real file.
Nobody wrote a fake file. It's mostly catch-all routing: the site sends every unknown address to the homepage. But any tool that checks "does this site have llms.txt?" by status code alone gets a false yes. If you've ever been told "your site already has one," check it yourself.
The files are well built. That's not the same as useful.
89% of real files rate Good or Strong on our structure score. The median is 77 out of 100.
- Opens with a title line
- 96%
- Has headed sections
- 96%
- Contains links
- 94%
- Has a one-line summary under the title
- 80%
- Also publishes an llms-full.txt
- 13%
- Links to .md versions of pages
- 5%
Only 5 of 128 files carry Yoast's signature, and 2 more come from documentation or generator tools. The other 95% were built in-house or by tools that don't sign their work. Big sites mostly don't run WordPress SEO plugins.
From files with no links at all (8 of them) to 3,772 links at salesforce.com. Docs-heavy companies like Datadog (2,407 links) and Unity (2,429) publish some of the largest.
Our score measures structure and size: does the file look the way the proposal describes? It can't judge whether the content is accurate, current or useful, and it can't tell you whether any AI tool reads it.
is how often sites with llms.txt appear in Google's AI Overviews, compared with similar sites without one. No difference. Our data rules out a lift bigger than about 9%.
Compare like with like, and the benefit disappears.
To test whether llms.txt helps, you need sites that are otherwise similar. The top 1,000 domains are too different from each other for that. So we used our Home Services Growth Index: 397 HVAC, plumbing, electrical and roofing companies that rank for at least 50 searches showing a Google AI Overview. For each, we measured how often the Overview lists its site as a source, and compared sites with and without llms.txt inside the same traffic group.
Look only at raw medians and llms.txt seems to help: sites with it score 58 on our Growth Index, against 52 without. But bigger companies are more likely to have the file, and they score higher anyway. Account for size and the gap vanishes. Most "it worked for us" stories skip that step.
llms.txt was designed for AI assistants and agents, not Google's search results. Chrome's Lighthouse tool added an llms.txt check in May 2026 for AI agents browsing a site.1 That's the likely reason developer-docs companies lead adoption: coding assistants read docs. We didn't test ChatGPT, Claude or Perplexity citations, and that's the next study worth running.
Match the effort to who's actually reading.
Skip it, or leave on whatever your plugin made. Don't pay for one. It won't move Google's AI answers.
Worth doing well. Your customers' AI tools read your docs. Write real descriptions and keep it current.
Ask for evidence it changed AI appearances, compared with similar sites that didn't add it.
Find out what's actually holding your site back.
We'll check your AI visibility the way we did here, and give you your top three fixes. No deck of fluff, no pressure.
Sample (adoption study). The top 1,000 domains on the Tranco list (GQG5K, October 19, 2026). Tranco blends DNS, traffic and link signals, so it's a popularity list, not strictly "most visited." 323 entries are infrastructure hostnames (ad servers, CDNs, APIs) and are reported separately. The rates above use the 620 websites that gave a clear answer.
What we couldn't reach. 57 websites blocked us or didn't respond. If none of them publish a file, adoption is 18.8%; if all of them do, 27.2%. Counting each company once (many companies own several domains), adoption is 25.1%.
Fetch and checks. Each domain was requested at both its bare and www address, for llms.txt and llms-full.txt, on October 20, 2026, from one US location with a standard browser identity. A file counts only if it returns a real text file at the llms.txt address, not a web page or redirect. Blocked results were re-checked with a second client and then a real browser, and a random control address caught sites that answer every request.
Categories and score. Site categories were assigned with AI assistance and spot-checked by hand. Categories under 30 sites are shown as counts only. The quality score (0 to 100) rewards following the proposal's structure, size and curation; it doesn't measure accuracy or usefulness. Some rules were tightened after seeing the data; every change and its effect is logged.
Effectiveness test. From the LVL3 Home Services Growth Index 2026: 397 companies with 50 or more AI Overview keywords, Semrush AI Overview data, October 20, 2026. Compared with traffic and trade held constant (ratio 0.92, 95% confidence interval 0.78 to 1.09). Google AI Overviews only. An association test, not an experiment.
1 TechWyse, "Google publishes first official AI search optimization guide, then Chrome starts auditing llms.txt," May 2026. techwyse.com
2 Search Engine Land, "Google says normal SEO works for ranking in AI Overviews and llms.txt won't be used," July 24, 2025. searchengineland.com
3 Search Engine Journal, "llms.txt shows no clear effect on AI citations based on 300K domains," November 20, 2025. searchenginejournal.com
Tranco list GQG5K: tranco-list.eu