How many websites have an llms.txt or AI crawler rules? We checked 474 in six industries

The short answer

116 of 472 sites published an llms.txt file, and 61 of 458 named any AI crawler in robots.txt. We checked design agencies, clinics and real estate agencies on 12 October 2026, and restaurants, schools and SaaS companies on 16 October. Every industry page linked below has its own sample, method and findings.

By industry

Software companies lead, schools trail. SaaS sites publish llms.txt at more than twice the rate of any other industry here, which fits a sector whose buyers and staff use AI tools daily and whose docs teams publish machine-readable files anyway. Schools are at the other end: 1 of 64 named any AI crawler, and 2 of 66 published llms.txt. Restaurants name AI crawlers about as often as SaaS companies, but for a different reason: 15 of the 18 restaurants that named one were built on Squarespace.

A rule is often a default

All 19 Squarespace sites in our data named the same list of AI crawlers, with the same rules. Squarespace's standard robots.txt lists more than 25 AI and data crawlers, from GPTBot and ClaudeBot to CCBot and Google-Extended, in the same group as every other bot, with the same rules: only account, search and system pages are off limits. So naming the crawlers changes nothing about what they can read.

That is 19 of the 56 sites in our checker's data that named any AI crawler, about a third. If you count "names an AI crawler" as a sign that a business has thought about AI, a third of the count is one platform's template. When you audit a site, read what the rule says, not whether a bot's name appears.

The names that matter also differ by job. OpenAI separates OAI-SearchBot, which serves ChatGPT search, from GPTBot, which collects training data. Google-Extended controls Gemini apps and grounding, not Google's AI Overviews, which follow normal Googlebot rules. The detail is in how AI decides which businesses to name and AI search is two games.

llms.txt follows the platform

Among the sites whose robots.txt we could read, llms.txt adoption tracked the website platform more than the industry.

Sites built on Webflow, HubSpot and Next.js published llms.txt far more often than WordPress or Squarespace sites. Some of that is who chooses each platform: many of the Next.js and HubSpot sites here are software companies, and many of the Webflow sites are design agencies. Either way, adoption follows what a platform makes easy, not evidence that the file works. We found the same pattern among design agencies in our platform comparison.

What actually matters

Presence of a file is not the same as anyone reading it. Ahrefs looked at about 38,000 sites with a valid llms.txt in May 2026 and found that 97% received no requests for the file at all. Google says that no special files or markup are needed to appear in its AI features, which use the same index as normal search. Our own reading of the evidence on llms.txt is in does llms.txt do anything?

What AI answers repeat is the content of your pages: who you are, what you offer, what it costs, and how to start. Across the industries we checked, the gaps that mattered were in those facts, such as fees that do not say which year they apply to, or opening hours written on the page but not in markup, not in optional files.

What to do

• Read your robots.txt. If your platform added AI crawler names, check what the rules actually allow before assuming you have blocked or invited anyone.
• Decide search and training separately. Allowing OAI-SearchBot while blocking GPTBot is a legitimate choice, and so is the reverse.
• Treat llms.txt as optional housekeeping. Publish one if your platform makes it easy and it describes your site accurately, but do not expect it to change what AI tools say.
• Put the effort into the facts buyers ask for, written plainly on pages that search engines can read.

How we checked

One checker, the same rules, six industries. For each website we fetched the homepage, robots.txt and the root llms.txt. A robots.txt "names an AI crawler" if it has a group for any of 11 AI crawlers, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. An llms.txt counts if the root address returns a text file starting with a heading, not a web page. For SaaS, a second study checked 60 products by hand with a wider list of 14 crawlers, and its results are combined with ours. Samples came from ordinary web searches in the UK, US, Australia and other English-speaking markets.

Limits

• These are search-based samples, not random ones. They lean towards established businesses that rank in search.
• We checked whether a file exists and what robots.txt says, not whether any crawler obeys it or reads the file.
• Root llms.txt only. A file elsewhere on a site, such as in a docs folder, is not counted.
• Platform figures mix industries, so they show association, not cause.