Which AI crawlers can read your site?
Enter your domain. We read your robots.txt the way each crawler does and show who's allowed in: AI training bots, AI search, AI assistants and search engines, plus the basics anyone can see from outside.
What this checks
- AI crawlers. For each of 14 AI bots, from GPTBot and ClaudeBot to PerplexityBot, Meta's crawler and Bytespider, whether your
robots.txtallows it to read your home page. They're grouped by what they do, because blocking a training bot and blocking an AI search bot have very different effects. - Search engines. Googlebot, Bingbot, Applebot and DuckDuckBot. If one of them is blocked, you'll see a warning at the top.
- Indexing. Whether your home page carries a
noindextag, and whether a sitemap exists and how many addresses it lists. - Security basics. HTTPS and the security headers browsers use to protect your visitors.
- Platform and CDN. What your site is built with and whether it sits behind a CDN such as Cloudflare.
The three kinds of AI bot
"Block AI" isn't one decision. AI companies run different bots for different jobs, each with its own name in robots.txt:
| Kind | What it does | Examples |
|---|---|---|
| Training | Collects pages to train future models. Doesn't link back. | GPTBot, ClaudeBot, CCBot, Bytespider, Meta's crawler, Google-Extended |
| AI search | Builds the index AI answers quote from, often with a link to you. | OAI-SearchBot, PerplexityBot, Claude's search crawler |
| Assistant | Fetches one page right now because a person asked about it. | ChatGPT-User, Claude-User, Perplexity-User |
Many site owners opt out of training and stay open to AI search and assistants, so they can still be cited. Our post on which AI crawlers read our sites has a ready-made robots.txt for that.
What this check can't tell you
robots.txt says who's welcome. It doesn't say who actually came, or whether they listened. Any software can call itself GPTBot, and some do: on our own sites, more than half of the requests claiming to be ChatGPT came from addresses OpenAI doesn't use. Only your server logs show which crawlers really read your pages, how often, and which ones were fakes. You can paste a sample of your log to see it, or check a single address with the crawler checker.
Questions
Does blocking GPTBot remove my site from ChatGPT?
Not completely. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches a page live when someone asks about it. Each has its own robots.txt name, so you can block one and allow the others.
Does robots.txt actually stop AI crawlers?
Only the ones that follow it. The major AI companies say their crawlers do, but robots.txt is a request, not a lock, and any software can claim a crawler's name. To see who really reads your pages you need your server logs.
Does blocking Google-Extended affect Google Search?
No. Google-Extended only controls whether your pages may be used for Google's AI models such as Gemini. Googlebot still crawls for Search, including AI Overviews.
What does the check read?
Your home page, robots.txt and sitemap, the same public files any visitor or crawler can read. It doesn't look for hidden files, and nothing is stored. Results are kept for 10 minutes so a repeat check is fast.
More free tools
Access log analyzer
Paste lines from your nginx or Apache log and see people, crawlers, fakes and attacks, sorted in seconds.
Free toolGooglebot checker
Check if an address claiming to be Googlebot, GPTBot or another crawler really belongs to it.
Related reading
AI crawlers read 9x more of our pages than Googlebot
ClaudeBot loaded 1,377 of our pages in 15 days; Googlebot loaded 430. Which AI bots read your site, what each one is for, and how to choose who gets in.
Security1 in 6 'Googlebots' on our sites was fake
Fake Googlebot, GPTBot and ChatGPT requests came from rented cloud servers, hunting for .env files. How to tell a real crawler from a fake in one minute.