llms.txt Reality Check methodology

By Matt Lawler, President and CEO, LVL3.

What we measured

We checked whether 1,000 of the web's highest-ranked domains publish an llms.txt file, and what the published files look like. We fetched every domain on October 20, 2026. This page explains each rule we used, every rule we changed after seeing the data, and the errors we found and fixed. Every number here comes from the final dataset.

The sample

• The list is Tranco list GQG5K, generated October 19, 2026: the top 1,000 entries, ranks 1 to 1,000.
• Tranco is a research ranking. It blends five sources (Chrome UX Report, Farsight DNS data, Majestic links, Cloudflare Radar and Cisco Umbrella) over 30 days. It isn't a list of the most-visited websites. It mixes DNS lookups, traffic and links, so it includes plenty of plumbing.
• Of the 1,000 entries, 323 are infrastructure hostnames (CDNs, ad servers, DNS and telemetry endpoints) and 677 are websites. See "What counts as a website" below.
• Two entries carry a www. prefix as Tranco published them. We fetched both the listed name and the bare name.

Our unit is a ranked domain, not a company

Each row is one ranked domain. Big companies appear many times. Among the 620 websites we could classify, Google operates 23 domains, Amazon 17 and Microsoft 17. Two published files are the same file reached from two ranked domains, so they're counted twice in the headline.

Because clusters can skew a rate, we also report adoption with one row per company. That rate is higher (25.1%) because the big clusters mostly don't publish a file. It matters most for the top 100. At the domain level, 7 of 64 top-100 websites publish a file (10.9%), against 21.6% for ranks 101 to 1,000 (Fisher p = 0.049). Collapsed to one row per company, it's 7 of 38 against 119 of 475, and the gap isn't significant (Fisher p = 0.44). So we say the top of the list isn't ahead. We don't say it adopts less.

How we fetched each domain

• Four GET requests per domain: /llms.txt and /llms-full.txt, on both the bare host and the www host.
• HTTPS, redirects followed (up to 10), 10-second timeout, a standard desktop Chrome user agent, certificate checks on.
• We stored the first 5 KB of every response and measured the full size up to 5 MB. For every file we classed as present, we fetched the full body again for scoring.
• One run, from one machine in the United States. We retried timeouts once.
• We added a control request for every candidate file: a random path like /llms-1a2b3c4d.txt on the same host. If the control path returned the same body, the host answers every path the same way, so we didn't count the file.
• Our first fetcher didn't decode compressed responses. We fixed that and re-fetched the affected files.

How we classified each response

Each host gets a status. Then the two hosts combine into one class for the domain.

• Present: a 2xx response, the final URL still ends in llms.txt, the body is non-empty plain text (not HTML), it contains a markdown heading or link, and the control path doesn't return the same body.
• Soft-404: a 2xx response that isn't a file. Usually an HTML page, a redirect to another path, an empty body, a short "not found" message, or a catch-all host.
• Absent: a 404 or 410.
• Blocked: a 401, 403 or 429, any other unusual 4xx, or a recognized bot wall (Cloudflare, Akamai, AWS WAF and similar challenge pages).
• Unreachable: DNS failure, TLS failure, timeout, refused connection, or a 5xx with no challenge page.

The domain takes the highest status from either host, in this order: present, soft-404, absent, blocked, unreachable.

Blocked and unreachable domains are left out of the adoption rate, because we can't tell whether they have a file.

"Blocked" means blocked our clients, not blocked everyone. Every domain our first client saw as blocked was re-checked with a second plain HTTP client and then a headless Chrome browser. A domain stays blocked only if all three were walled out. That re-check found real files behind walls and moved 43 domains out of blocked.

What counts as a website

We tag each domain as website or infrastructure with a coded rule, run the same way on all 1,000 rows. A domain is a website when either host returns a 2xx HTML page with a real title (not a bot-wall title). That includes a redirect to a parent company's site.

When neither host gives a usable answer (no response, a block, an odd status, or a page with no title), the rule can't decide. We then fall back on what's publicly known about the brand. That judgment applies to 480 of the 1,000 tags. It's a big share, mostly because many backend hostnames don't answer at all.

The rule has one known rough edge. A tracking or CDN hostname that redirects to its parent's homepage counts as a website on that technicality.

Sector categories (12 of them) were assigned with the help of a language model working from general knowledge of each brand and the homepage title. We re-checked two random samples (80 rows in total) against the written rules and found and fixed one error. We print counts, not percentages, for any category with fewer than 30 websites.

The adoption rate, and the alternatives

Headline: 127 of 620 classifiable websites publish an llms.txt (20.5%, 95% CI 17.5% to 23.8%). All intervals on this page are Wilson score intervals.

The same data under other reasonable definitions:

• Duplicates removed: 125 of 618 (20.2%).
• Strict, file must live on the ranked domain: 111 of 620 (17.9%). 16 website files are reached through a redirect to another domain, such as a legacy brand domain pointing to the current one. The headline counts them, which adds 2.6 points.
• One row per company: 125 of 498 companies (25.1%).
• All 1,000 domains, infrastructure included: 128 of 681 (18.8%).

What the excluded sites could do to the rate. 57 websites stayed blocked or unreachable. If none of them publish a file, adoption is 18.8%. If all of them do, it's 27.2%. If each one adopts at its own category's rate, it's 20.5%, the same as the headline.

Rules we changed after seeing the data

We set the rules before fetching. Some didn't survive contact with real responses. Every change is listed here with its direction.

1. Absent narrowed to 404 and 410. We first counted any clean 4xx as absent. Codes like 400, 418 and 498 turned out to be servers rejecting the client, not answering about the file. They now count as blocked. When first applied, this moved 15 domains (9 websites) and raised the website rate from 21.3% to 21.7%. After the later re-checks, 6 domains (3 websites) remain blocked under this rule.
2. GitBook detection tightened. Our first fingerprint matched any file whose links end in .md. That flagged 16 files, none of which mention GitBook. A file now counts as GitBook only if it names GitBook or is served from a GitBook host. This changed no adoption counts. We report the .md link pattern separately (7 of 128 files).
3. Cross-domain redirects kept as present. A file reached through a redirect to another domain still counts. We report the strict rate above so you can choose.
4. A second quality score (v2). Our first score (v1) put two-thirds of files in the top band. We designed v2 after seeing that. Both are reported below.
5. Classifier corrections. We added the control path, the content check, compression decoding, tighter bot-wall detection and the three-client re-check of blocked domains. Net effect: 2 catch-all hosts reclassified as soft-404, 8 real files added, one file re-scored from Poor to Good, and dozens of domains moved between blocked, absent and soft-404.

Soft-404s are routing, not files

149 domains (136 websites, 21.9% of classifiable websites) answer /llms.txt with a 200 and something that isn't a file. That's catch-all routing in a framework or CDN. It says nothing about intent.

• Homepage or app shell: 134
• Branded 404 page served with a 200: 8
• Empty body: 3
• Other HTML page: 2
• Catch-all host (failed the control path): 2

22 of the 149 have a clean 404 on one host. They're soft-404 only because the other host's 200 wins under our precedence rule. Bot walls aren't in this group. We moved them to blocked.

The quality score

The score measures structure and size. It doesn't measure accuracy, freshness, or whether any AI system reads the file.

• v2 (headline): spec basics, 25 points (title line, summary line right after it, a section with links); scale, 35 points on log curves for links, sections and size; curation, 25 points (described links, an Optional section, sections that hold links); effort, 15 points.
• Every file that isn't an untouched plugin default gets 10 effort points automatically. Most files get them.
• Bands: Poor under 30, Basic 30 to under 55, Good 55 to under 80, Strong 80 and up. A 79.9 is Good.
• v2 uses full bodies. For 7 files the full re-fetch failed, so their v2 is a floor.
• Medians and quartiles use inclusive quantiles.

Side by side, on 128 files:

• v1: Poor 0, Basic 11, Good 30, Strong 87. Median 84.5.
• v2: Poor 5, Basic 9, Good 71, Strong 43. Median 77.1.

Basic shape: 123 of 128 open with a title line (96.1%), and 103 follow it with a summary line (80.5%). One file is a robots-style opt-out policy at the llms.txt path, and two use a non-spec format. We count all three as present, because a file exists.

Generator detection

We detect three WordPress plugins (Yoast, Rank Math, All in One SEO) and three docs platforms (Mintlify, GitBook, Docusaurus) by their credit lines, confirmed against plugin source or vendor docs. We found 5 Yoast files, 1 Mintlify file, and 1 file crediting a third-party generator we added after seeing it.

The other 121 are "Other or unknown". That bucket holds hand-written files and any tool we don't fingerprint. None of the 5 Yoast files is an untouched default. We didn't measure how many sites in the sample run WordPress, so low plugin counts say more about the sample than about the plugins.

Verification and known errors

• Present: a real browser loaded every present file. That check confirmed a non-HTML body existed. It didn't confirm the body was an llms.txt, so it missed 2 catch-all hosts. The control path caught them later. We also re-checked fields on 20 random files and found no errors.
• Absent: we browser-checked 100 random absent domains and found no grading errors. For three of them the browser got no answer, so those three are unverified.
• Soft-404: 50 random domains were checked by hand from screenshots. None hid a real file. Two were bot walls, a class of error we then fixed across the dataset.
• Blocked: a 30-domain sample was wrong 23 times (77%). That breached our 5% re-grade threshold, so we re-graded every blocked domain with two more clients.

Limits

• One run, one US location, one moment. Regional routing and transient errors stick.
• Our clients aren't human visitors. Some blocked sites would likely serve a person.
• Category and website tags involve judgment, disclosed above.
• Nothing here measures whether AI systems read or reward llms.txt.

Does llms.txt help? The effectiveness test

This test uses a different dataset: the LVL3 Home Services Growth Index 2026.

• Sample: 397 home services companies with 50 or more AI Overview keywords.
• Data: Semrush AI Overview data, snapshot October 20, 2026. Google AI Overviews only.
• How we detected llms.txt: during the Growth Index crawl on October 20, 2026, we requested /llms.txt on each site. A site counts as having one when the request returns a 200 response with a body that starts with a # heading mark and isn't HTML. See the Growth Index methodology for the crawl.
• Outcome: the share of eligible AI Overviews that cite the company's site, modeled as the log of a smoothed rate.
• Model: OLS with HC3 robust standard errors, controlling for log monthly organic traffic and trade.
• Result: sites with an llms.txt had a citation rate 0.92 times that of sites without one (95% CI 0.78 to 1.09, p = 0.32). That's no detectable difference.
• Raw medians: 4.7% with a file (n = 104) against 4.4% without (n = 293).

This is an association, not an experiment. It can't rule out a small effect, and it says nothing about AI systems other than Google AI Overviews.

Reproduce it

Download Tranco list GQG5K (top 1,000). Request the four URLs per domain with the settings above. Apply the classification rules and precedence. Add the random control path. Contact us for the per-domain results.

Tranco list GQG5K: tranco-list.eu

Back to the report