The short answer
Your website can be blocking AI crawlers in a place nobody looks. In the LVL3 Home Services Growth Index, under 3% of the 464 sites we crawled on October 20, 2026 blocked an AI crawler in robots.txt, the file that tells crawlers what they may read. But up to 8% had servers that refused an AI crawler's request. Reading robots.txt alone can tell you you're open when you aren't.
Here's what we found, which blocks actually cost you, and how to check your own site.
What we found across 464 home services sites
We checked each site's robots.txt rules for 11 AI crawlers, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. Then we requested each homepage as a normal browser and as the major AI crawlers, from more than one network.
- 13 of 464 sites (2.8%) block an AI crawler in robots.txt.
- Up to 37 of 464 (8.0%) had a server that refused at least one AI crawler.
- 10 of 464 (2.2%) refused clearly across multiple bots. That's the firmest number we have.
Why the 8% is a ceiling
A request that only claims to be GPTBot can trip anti-spoofing rules that the real, verified bot would pass. That's why we report the server number as "up to". The point holds either way: a refusal at the server never shows up in robots.txt, so the file can look clean while the door is shut.
Where to look for a server block
From outside, we can see a refusal but not its cause. So treat these as places to look, not causes we measured:
- A firewall or CDN bot rule. Since July 1, 2025, Cloudflare asks every new domain at sign-up whether to allow AI crawlers, and it blocks them by default (DBTA, July 1, 2025).
- A security plugin with bot rules that catch AI user agents along with the spam.
Blocking AI crawlers: which blocks actually cost you
Not every AI crawler does the same job, so not every block costs the same.
- OAI-SearchBot is the crawler behind ChatGPT search. OpenAI says sites that opt out of it won't be shown in ChatGPT search answers. Block this one and you're out of those answers.
- GPTBot is for training. OpenAI says disallowing it signals that your content shouldn't be used to train its models. Blocking it is a business choice, and a reasonable one.
- Google-Extended doesn't control AI Overviews. Google says robots.txt rules for Googlebot are the control for its AI search features. Google-Extended limits training and grounding in some of Google's other systems.
A choice versus an accident
If you've decided to opt out of training, block GPTBot on purpose, in robots.txt, and leave the search crawlers open. That's a decision you can defend. A firewall rule nobody remembers setting isn't.
How to check yours
- Open yourdomain.com/robots.txt. Look for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended above a Disallow line.
- Request your homepage as a crawler. In a terminal, run curl -A "OAI-SearchBot" -I https://yourdomain.com. A 200 means it got in. A 403 or 410 means something refused it.
- Treat a refusal as a lead, not a verdict. Your firewall may let the verified bot through. Open its bot rules, and its AI crawler setting if you're on Cloudflare, and see what's switched on.
- Fix it, then check again. OpenAI says it can take about 24 hours after a robots.txt change for its search systems to adjust.
How we handle it
Crawl access sits at the start of our SEO work. Crawl, schema and internal-link fixes land in the first 30 days, then get re-audited every month so they don't quietly break. A regional home services operator we work with got a rebuilt site that's crawlable by default, and it owns that site outright.
Sources
- The LVL3 Home Services Growth Index 2026: 464 operator websites crawled October 20, 2026. Server refusal of AI crawlers is an upper bound; the hard-block count is a clear refusal across multiple bots.
- OpenAI, "Overview of OpenAI Crawlers," developers.openai.com/api/docs/bots, accessed November 13, 2026.
- Google Search Central, "AI features and your website," last updated December 10, 2025.
- DBTA, "Cloudflare Rolls Out Setting to Automatically Block AI Companies from Collecting Users' Digital Data," July 1, 2025.