AI crawlers read 9x more of our pages than Googlebot
In 15 days, AI crawlers loaded 3,832 pages from our five websites. Search engines loaded 2,414, and Googlebot just 430. ClaudeBot alone read more than three times as many pages as Googlebot. Here's who's reading, what each bot is for, and how to decide who gets in.
For twenty years, "who crawls my site" had a simple answer: Google, then Bing, then a long tail. When we sorted every request to our five websites from 2 to 16 October 2026, that answer was out of date. Most of the pages fetched by crawlers were fetched for AI.
The crawlers, ranked by pages loaded
| Crawler | Run by | Kind | Pages |
|---|---|---|---|
| PetalBot | Huawei | Search | 1,717 |
| ClaudeBot | Anthropic | AI | 1,377 |
| Meta AI crawler | Meta | AI | 567 |
| Amazonbot | Amazon | AI | 523 |
| Googlebot | Search | 430 | |
| GPTBot | OpenAI | AI | 321 |
| PerplexityBot | Perplexity | AI | 310 |
| GoogleOther | AI and research | 295 | |
| OAI-SearchBot | OpenAI | AI | 277 |
| Bytespider | ByteDance | AI | 144 |
| Bingbot | Microsoft | Search | 83 |
Page loads (HTML pages, not images or files), our five sites, 2–16 Oct 2026. Googlebot, Bingbot, GPTBot, OAI-SearchBot and PerplexityBot were checked against the address ranges their owners publish; fakes are left out.
Leave out PetalBot, which crawls heavily for Huawei's own search engine, and the gap gets wider: Google and Bing together loaded 513 pages, while the AI crawlers loaded 3,832, more than seven times as many.
Not all AI bots want the same thing
"AI crawler" covers three different jobs, and the difference matters when you decide what to allow:
- Training crawlers collect pages to train future models: GPTBot, ClaudeBot, Meta's crawler, Bytespider, Common Crawl's CCBot. What they read may shape what a model "knows", but they don't link back to you.
- AI search crawlers build an index that AI answers quote from, often with a link: OAI-SearchBot for ChatGPT search, PerplexityBot, Claude's search crawler. These are the closest thing to Googlebot for AI answers.
- AI assistants fetch one page right now, because a person asked about it: ChatGPT, Claude and Perplexity's user agents, DuckDuckGo's AI, Meta AI. We saw 362 of these requests, led by ChatGPT (94), Claude (89) and DuckDuckGo's AI (86). Each one is a person, somewhere, asking an AI about something on your site.
They do read robots.txt
AI crawlers asked for our robots.txt 455 times in the period. The major ones say they follow it, and each has its own name you can allow or block. Google is different: Googlebot does the fetching for both search and Google's AI, and a separate Google-Extended line decides whether your pages may be used for Gemini. Blocking it doesn't affect Google Search.
How to choose
A common middle path: stay visible in AI answers, but opt out of training.
# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /
# AI search and live fetches stay allowed (they can cite and link to you)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /Check each company's documentation for the current names before you copy this; they change. And remember that robots.txt is a request, not a lock. Only well-behaved bots follow it, and software can claim to be any of these names. On our sites, more than half of the requests claiming to be ChatGPT were fake. Here's how to tell.
What to watch on your own site
- Which pages AI reads. If your best guides never get fetched, AI answers can't draw on them.
- Live fetches from assistants. They show which pages people ask AI about, a signal you won't find in Search Console.
- What it costs. Crawlers that load thousands of pages use real bandwidth and server time. Decide if what you get back is worth it.
Humansift shows every AI crawler and assistant per page, verified against published address ranges, next to your Google Search and Analytics data.
How we measured
Every number here comes from the server logs of our own five websites (a pet guide, an education site, a small-business tool, a crafts marketplace and a fashion blog), from 2 to 16 October 2026: 400,425 page and API requests. Images, styles and scripts were counted separately and are left out. Each request was sorted by Humansift: named bots by their user agent, Googlebot, Bingbot, GPTBot, ChatGPT and Perplexity checked against the address ranges their owners publish, attack paths by pattern, and data-centre traffic by network. A "visitor" is one address and browser on one day. Five small sites are a small sample, so read these as one honest data point, not an industry average.