- AI Crawlers
- generative Engine Optimization
AI Crawlers List: GPTBot, ClaudeBot, PerplexityBot and Whether to Block Them
Every AI crawler that matters, from GPTBot and OAI-SearchBot to ClaudeBot, PerplexityBot and Google-Extended: what each one does, which to allow, which you can block, and robots.txt templates.
Collins
On this page
- Key takeaways
- The three jobs AI crawlers do
- The full AI crawler list
- Which AI crawlers to allow, and which to block
- What blocking does, and what it doesn't
- Your CDN may be blocking AI crawlers for you
- How much do AI crawlers give back?
- Robots.txt templates
- Make sure crawlers can read what they reach
- How to check what's reaching your site
- FAQ
- What is an AI crawler?
- What is GPTBot?
- Should I block AI crawlers?
- Does blocking Google-Extended remove me from AI Overviews?
- Do AI crawlers respect robots.txt?
- How do I check if AI crawlers can access my site?
Every major AI company now sends bots to your website. Some collect content to train AI models. Some build the search index that ChatGPT, Claude and Perplexity pull answers from. Others fetch a single page because someone just asked a question about it.
Blocking the wrong one can quietly remove your brand from AI answers. This guide lists every AI crawler that matters, explains what each one does, and shows which to allow and which you can safely block.
Key takeaways
- AI crawlers do three different jobs: training models, building AI search indexes, and fetching pages for a user in real time. Each job uses a different bot.
- Search bots decide whether you appear in AI answers. Block OAI-SearchBot, Claude-SearchBot or PerplexityBot and you disappear from that tool's answers.
- Training bots decide whether future models learn from you. Blocking them doesn't affect today's search answers.
- You can't opt out of Google's AI Overviews without opting out of Google Search. They both run on Googlebot.
- Many sites block AI crawlers without knowing it, through old robots.txt rules or CDN settings. Check yours before anything else.
The three jobs AI crawlers do
Every AI crawler falls into one of three groups. Knowing which group a bot belongs to tells you what happens if you block it.
| Type | What it does | Examples | If you block it |
|---|---|---|---|
| Training crawler | Collects content to train future AI models | GPTBot, ClaudeBot, CCBot, meta-externalagent | Future models learn less about you. Today's AI answers are unaffected |
| Search crawler | Builds the index an AI tool searches when it answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot | Your pages stop appearing as sources in that tool's answers |
| User-triggered fetcher | Visits a page because a user asked about it | ChatGPT-User, Claude-User, Perplexity-User | The AI can't read your page live, even when someone asks for it |
Training still makes up most AI crawling. In Cloudflare's analysis of traffic across its network, 79% of AI crawling in July 2025 was for training, 17% for search and about 3% for user actions. User-triggered fetching is the fastest-growing type: Cloudflare's 2025 year in review found it grew more than 15 times between January and December.
The full AI crawler list
These are the crawlers that matter for brand visibility, grouped by company. The robots.txt token is the name you use in your robots.txt file.
| Company | Robots.txt token | Type | What it powers |
|---|---|---|---|
| OpenAI | [GPTBot](https://developers.openai.com/api/docs/bots) | Training | Training OpenAI's models |
| OpenAI | OAI-SearchBot | Search | ChatGPT search results |
| OpenAI | ChatGPT-User | User-triggered | Live page visits from ChatGPT and custom GPTs |
| Anthropic | [ClaudeBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Training | Training Claude models |
| Anthropic | Claude-SearchBot | Search | Claude's web search results |
| Anthropic | Claude-User | User-triggered | Live page visits when a Claude user asks |
| Perplexity | [PerplexityBot](https://docs.perplexity.ai/guides/bots) | Search | Perplexity's search index (not used for training) |
| Perplexity | Perplexity-User | User-triggered | Live page visits for Perplexity answers |
| [Googlebot](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) | Search | Google Search, AI Overviews and AI Mode | |
| Google-Extended | Control token only | Use of your content for Gemini training and grounding | |
| Microsoft | Bingbot | Search | Bing and Microsoft Copilot |
| Apple | [Applebot](https://support.apple.com/en-us/119829) | Search | Siri, Spotlight and Safari, and may be used for Apple's AI models |
| Apple | Applebot-Extended | Control token only | Use of your content for Apple's AI model training |
| Meta | [meta-externalagent](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) | Training | Training Meta's AI models and indexing content |
| Meta | meta-externalfetcher | User-triggered | Live page visits for Meta AI |
| Amazon | [Amazonbot](https://developer.amazon.com/amazonbot) | Training and products | Amazon services, may be used to train Amazon's AI models |
| Amazon | Amzn-SearchBot | Search | Search in Amazon products such as Alexa |
| Amazon | Amzn-User | User-triggered | Live answers in Alexa |
| Common Crawl | CCBot | Training | An open web archive used to train many AI models |
Two of these are not crawlers at all. Google-Extended and Applebot-Extended are control tokens. Googlebot and Applebot do the crawling, and these tokens only decide how the content they collect may be used for AI.
Which AI crawlers to allow, and which to block
The right setup depends on what you want from AI tools. For most marketing, SaaS and ecommerce sites, the goal is to be recommended, so search and user-triggered bots should always be allowed.
| Your goal | Search crawlers | User-triggered fetchers | Training crawlers |
|---|---|---|---|
| Be recommended in AI answers (most brands) | Allow | Allow | Allow |
| Appear in AI answers but keep content out of training | Allow | Allow | Block |
| Protect paid or licensed content | Allow selectively | Review case by case | Block |
Why we recommend most brands allow training crawlers too: AI models answer many questions from what they learned in training, without searching. If your content was never in the training data, the model knows less about your brand, your category and how you compare. For a business that wants to be recommended, that's a cost with little upside. Publishers that sell their content are the main exception.
Search crawlers are the ones you can't afford to block. Blocking OAI-SearchBot removes you from ChatGPT search answers. Blocking Claude-SearchBot or PerplexityBot does the same in Claude and Perplexity. Blocking Googlebot removes you from Google entirely.
What blocking does, and what it doesn't
Robots.txt is a request, not a lock. Before you change anything, it helps to know its limits.
- Blocking a training crawler doesn't remove you from existing models. It only keeps your new content out of future training.
- User-triggered fetchers may not follow robots.txt. ChatGPT-User, Perplexity-User and meta-externalfetcher can visit a page when a user asks for it, because a person, not a crawler, started the request.
- You can't block AI Overviews on their own. AI Overviews and AI Mode use Googlebot, and Google-Extended doesn't change your inclusion in Google Search. To limit what Google shows from a page in AI features, use nosnippet or max-snippet on that page.
- Changes take time. OpenAI, Perplexity, Meta and Amazon all process robots.txt changes in about 24 hours.
- Each subdomain needs its own rules. A robots.txt on www.example.com doesn't cover blog.example.com.
Your CDN may be blocking AI crawlers for you
Many sites block AI crawlers without anyone on the marketing team deciding to. The two usual causes:
- CDN and firewall settings. In July 2025, Cloudflare switched to blocking AI crawlers by default and launched paid access for crawlers. Other security tools have similar bot settings. If your site runs behind a CDN, check its bot and AI crawler settings, not just robots.txt.
- Old robots.txt rules. Many sites added blanket rules in 2023 and 2024 to block "AI bots". Those rules often caught search crawlers that didn't exist yet, or block entire companies rather than single bots.
Cloudflare's 2025 review found that GPTBot, ClaudeBot and CCBot were the most fully blocked crawlers among the top 10,000 domains, as Search Engine Journal reported. If your brand wants AI visibility, make sure you're not one of them by accident.
How much do AI crawlers give back?
AI crawlers take far more than they send back in traffic, and the gap varies widely by company. Cloudflare measured how many pages each company crawled for every visit it referred back to websites in July 2025:
| Company | Pages crawled per referred visit |
|---|---|
| Anthropic | about 38,000 |
| OpenAI | about 1,100 |
| Perplexity | about 195 |
| about 5 |
These ratios explain why some publishers block training crawlers. For brands, the calculation is different. The value of AI visibility comes from being named and recommended in answers, which often happens without a click. Our guide to AI referral traffic shows how to measure the visits that do come through.
Robots.txt templates
Allow all AI crawlers (recommended for most brands). If your robots.txt has no rules naming these bots, they're already allowed. Adding them explicitly protects against broad rules elsewhere in the file:
User-agent: OAI-SearchBotUser-agent: ChatGPT-UserUser-agent: GPTBotUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: ClaudeBotUser-agent: PerplexityBotUser-agent: Perplexity-UserAllow: /Appear in AI answers, but keep content out of training:
User-agent: GPTBotUser-agent: ClaudeBotUser-agent: CCBotUser-agent: meta-externalagentUser-agent: Google-ExtendedUser-agent: Applebot-ExtendedDisallow: / User-agent: OAI-SearchBotUser-agent: ChatGPT-UserUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: PerplexityBotUser-agent: Perplexity-UserAllow: /Keep your existing rules for Googlebot and Bingbot as they are. Test any change before you publish it, because one wrong line can block your whole site.
Make sure crawlers can read what they reach
Allowing a crawler is only half the job. Most AI crawlers read the raw HTML your server sends and don't run JavaScript. If your key content, pricing or product details load with JavaScript, AI crawlers may see an empty page. We cover this in AI Crawlers Do Not Render JavaScript.
How to check what's reaching your site
- Check your robots.txt rules. Our free AI Crawler Checker shows which AI and search crawlers your homepage allows and the rules behind each result.
- Check your CDN and firewall. Look for AI crawler or bot-blocking settings in Cloudflare, Akamai, Fastly or your hosting dashboard.
- Read your server logs. Search for the bot names in the user-agent field to see which crawlers visit, how often, and which pages they read.
- Verify the real bots. Anyone can fake a user agent. OpenAI, Anthropic and Perplexity publish the IP addresses their bots use, so you can confirm a visit is genuine.
For a full technical review, use the scored checklist in our AEO audit guide.
FAQ
What is an AI crawler?
An AI crawler is a bot that visits websites for an AI company. Some collect content to train AI models, some build the search index AI tools use to answer questions, and some fetch a page in real time when a user asks about it.
What is GPTBot?
GPTBot is OpenAI's training crawler. It collects public web content that may be used to train OpenAI's models. Blocking it doesn't remove you from ChatGPT search answers. That's controlled by a separate bot, OAI-SearchBot.
Should I block AI crawlers?
Most brands shouldn't. Blocking search crawlers removes you from AI answers, and blocking training crawlers means future AI models know less about your brand. Blocking training crawlers makes sense mainly for publishers that sell or license their content.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls whether your content is used to train and ground Gemini models. It doesn't affect inclusion in Google Search, and AI Overviews and AI Mode use Googlebot. To limit what appears from a page in AI Overviews, use nosnippet or max-snippet.
Do AI crawlers respect robots.txt?
The major training and search crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta and Amazon follow robots.txt. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not, because a person started the request.
How do I check if AI crawlers can access my site?
Check your robots.txt with a tool such as Lantern's AI Crawler Checker, review your CDN's bot settings, and search your server logs for the bot names. Confirm visits are genuine by checking them against the IP lists that AI companies publish.