1 in 33 local business sites refuse an AI search bot their robots.txt allows, theStacc study finds

The AI Lockout Index, November 2026: 1 in 33 US local business websites refuses requests identifying as an AI search bot that its own robots.txt allows. Source: theStacc.

Glastonbury, Connecticut, United States – November 9, 2026 – One in 33 US local business websites refuses requests identifying as an AI search bot that its own robots.txt file allows, while serving requests identifying as Googlebot, according to The AI Lockout Index, a new study by theStacc. The figure is 3.0%: 2,523 of the 83,504 sites whose robots.txt allows Googlebot and at least one AI search bot. A robots.txt file is where a website publicly lists which crawlers may visit; the AI search bots tested are the crawlers ChatGPT, Claude and Perplexity use to find and cite web pages in their answers.

Complete study results, with breakdowns by industry, state and website builder, 14 charts and downloadable data (no individual business is named): https://thestacc.com/research/ai-crawler-blocking-statistics/

1 in 33 local business websites refused requests identifying as an AI search bot their robots.txt allows (2,523 of 83,504). Source: The AI Lockout Index, theStacc.

Of those 2,523 sites, 2,100 (83.2%) returned a block response matching at least one other site’s, with the same status codes, server, page size and title. That pattern is consistent with shared platform or hosting rules rather than decisions by each owner. The 10 largest groups of matching set-ups account for 58.6% of the lockouts.

83.2% of these lockouts return a block page matching other sites’; the largest group is 949 health-practice sites on one platform. Source: The AI Lockout Index, theStacc.

theStacc requested the homepages of 140,000 US local businesses, drawn from Overture Maps Places across 30 industries, all 50 states and Washington, D.C., between October 31 and November 2, 2026. Each homepage was requested as a desktop browser, as Googlebot and as eight AI bots from OpenAI, Anthropic and Perplexity; refusals were rechecked at least 24 hours later, and 88,640 sites loaded normally and were valid after parked and placeholder domains were removed. The requests identified as each bot but came from theStacc’s servers, so sites that verify crawlers by IP address may treat the real bots differently.

“A robots.txt file can say yes while the server says no,” said Siddharth Gangal, founder and CEO of theStacc. “Most of the block pages we found matched those of many other sites, which points to a platform or hosting setting. Owners who want to appear in AI answers should check what their server sends back, not only what their robots.txt file says.”

Other findings from the study:

• One shared practice-website platform accounts for 37.6% of the 2,523 lockouts: Of 967 websites sharing one robots.txt template, used mainly by chiropractors, dentists, veterinarians and optometrists, 949 (98.1%) refused requests identifying as GPTBot and all three Claude bots, although the template allows GPTBot and ClaudeBot by name. Requests identifying as OAI-SearchBot, PerplexityBot and Googlebot were served on all 967.
• Square’s builders refuse ClaudeBot but serve AI search bots: 98.5% of Square Online sites (852 of 865) and 94.6% of Weebly sites (2,220 of 2,346) refused requests identifying as an AI training crawler, almost always Anthropic’s ClaudeBot, while serving requests identifying as every AI search bot. Requests identifying as GPTBot were served on more than 95% of these sites. Weebly is part of Square, so the groups overlap.
• robots.txt blocks are rarer than survey answers suggest: 7.1% of 98,195 local business websites (6,965) block at least one AI crawler in robots.txt; this base counts sites whose robots.txt request returned a file, a 404 or a web page. In a Clutch survey published in September 2025, 57% of small businesses said they restrict AI crawlers. The samples and measures differ.
• Counting automated challenges, up to 13.6%: Including automated “Checking Your Request” challenges (HTTP 429) and sites that also turned away requests identifying as Googlebot, up to 12,058 of 88,640 sites (13.6%) turned away a request identifying as an allowed AI search bot. 61.3% of those sent only challenges, which may not apply to the real crawlers. On Wix, 79.0% of sites (7,136 of 9,035) challenged these requests, and 68.5% of those also challenged Googlebot requests. Outside Wix, the rate was 6.2%.
• Most sites are open: 78.8% of sites (69,821 of 88,640) served requests identifying as all three AI search bots normally.
• 1 in 10 homepages shows almost no text before JavaScript runs: 9.6% of homepages (8,547 of 88,640) contain fewer than 100 words of visible text in the HTML a crawler receives. Crawlers that do not run JavaScript see only that HTML.

Results by industry (highest and lowest five of 30 industries)

Any refusal includes automated challenges. On the stricter measure, dentists (7.1%) and nail salons (6.9%) follow chiropractors. Chiropractor, dentist, veterinarian and optometrist figures include the shared health-practice platform.

Chiropractors (23.9%) and physical therapists (21.4%) most often turned away requests identifying as AI search bots. Source: The AI Lockout Index, theStacc.

Results by state (highest and lowest five of 50 states and Washington, D.C.)

AI search lockout rates by state range from 11.3% in Tennessee to 19.9% in Alaska. Source: The AI Lockout Index, theStacc.

Results by website builder (builders with 1,000 or more sites tested; any refusal, including automated challenges)

More charts from the study

On one health-practice platform, GPTBot and Claude’s bots were served on 1.9% of 967 sites; OAI-SearchBot, PerplexityBot and Googlebot on all of them. Source: The AI Lockout Index, theStacc.

Square Online (98.5%) and Weebly (94.6%) sites refused requests identifying as ClaudeBot while serving AI search bots. Source: The AI Lockout Index, theStacc.

57% of small businesses say they restrict AI crawlers (Clutch, 2025); 7.1% of local business websites block any AI bot in robots.txt. Samples and measures differ. Source: The AI Lockout Index, theStacc.

Of 12,058 sites that turned away AI search bots, 61.3% sent only automated challenges; 20.9% refused them while serving Googlebot. Source: The AI Lockout Index, theStacc.

78.8% of US local business websites served the ChatGPT, Claude and Perplexity search bots normally. Source: The AI Lockout Index, theStacc.

9.6% of local business homepages have fewer than 100 words of text before JavaScript runs. Source: The AI Lockout Index, theStacc.

Summary of the study’s key findings. Source: The AI Lockout Index, theStacc.

Media pack (high-resolution charts, data files and method summary): https://thestacc.com/research/ai-lockout-index/ai-lockout-index-media-pack.zip

Journalists and publishers may use the charts and data with a link to the study.

About theStacc

theStacc is an AI marketing platform for local and multi-location businesses. Its AI writing tool writes SEO blog content, its SEO automation tool plans, scores and publishes that content, and its AI local SEO tool manages Google Business Profile posts and review replies. theStacc also schedules social media posts, and its free AI Visibility Checker shows whether ChatGPT and Claude can read a website. theStacc was founded in 2024 by Siddharth Gangal.

Media Contact
Company Name: theStacc
Contact Person: Siddharth Gangal
Email: Send Email
Phone: 860-758-8094
Address:2389 Main St, Ste 100
City: Glastonbury
State: Connecticut
Country: United States
Website: https://thestacc.com