JW2D OÜ

Which AI crawlers to allow in robots.txt: GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended and more

Yakiv Romanchuk, founder of JW2D · Published

Short answer

AI companies use separate crawlers for search and for training. To appear in AI answers, allow the search ones: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Bingbot (Bing and Copilot) and Googlebot (Google Search, AI Overviews, AI Mode). Training crawlers — GPTBot, ClaudeBot and the Google-Extended token — are a separate choice; Google states that Google-Extended does not affect inclusion or ranking in Google Search. Also check your CDN: Cloudflare has blocked AI crawlers by default since 1 July 2025.

AI crawlers and what they do

Company Agent Purpose (official documentation) For AI answers?
OpenAI OAI-SearchBot Surfaces websites in ChatGPT search results; opted-out sites are not shown in ChatGPT search answers Yes — allow
OpenAI ChatGPT-User Visits pages for certain user actions in ChatGPT and custom GPTs Yes — usually allow
OpenAI GPTBot Crawls content that may be used to train OpenAI foundation models No — training only
Anthropic Claude-SearchBot Indexes content to improve Claude’s search results; disabling it may reduce visibility Yes — allow
Anthropic Claude-User Fetches pages when users ask Claude questions; disabling it may reduce visibility in user-directed search Yes — allow
Anthropic ClaudeBot Collects web content that could contribute to model training No — training only
Perplexity PerplexityBot Surfaces and links websites in Perplexity search results; not used for training foundation models Yes — allow
Perplexity Perplexity-User Fetches pages for user questions; generally ignores robots.txt; not used for training Yes
Google Googlebot Google Search crawler; AI Overviews and AI Mode use pages indexed by Google Yes — allow
Google Google-Extended Token controlling use of content for training Gemini models and grounding in Gemini Apps and Vertex AI; does not affect Google Search inclusion or ranking Your choice
Microsoft Bingbot Bing crawler; Copilot grounds answers in Bing; Bing advises not to block Bingbot in robots.txt Yes — allow

How to decide

  • You want to appear in AI answers — allow all search agents in the table: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot.
  • You do not want your content used for training — you can disallow GPTBot and ClaudeBot and the Google-Extended token. According to the companies’ documentation, this is separate from their search crawlers.
  • You want maximum presence in AI — allow both. This is a business decision; there is no official statement that allowing training crawlers improves visibility in AI answers.

Example robots.txt

A minimal configuration that allows AI search and blocks training could look like this:

  • User-agent: OAI-SearchBot / Allow: /
  • User-agent: Claude-SearchBot / Allow: /
  • User-agent: PerplexityBot / Allow: /
  • User-agent: GPTBot / Disallow: /
  • User-agent: ClaudeBot / Disallow: /
  • User-agent: Google-Extended / Disallow: /

On jw2d.com we allow both search and training crawlers, because for us maximum presence in AI matters more. Perplexity notes that changes may take up to 24 hours to be reflected.

The Cloudflare trap

robots.txt is not the only gate. On 1 July 2025 Cloudflare announced that it was changing its default to block AI crawlers unless they pay creators for their content. In a post of 1 July 2026 it described new defaults from 15 September 2026 for new domains: Search bots stay allowed, while Training and Agent bots are blocked by default on pages that display ads; non-verified bots remain blocked by default.

If your site is behind Cloudflare or another CDN with bot protection, check its AI bot settings in addition to robots.txt. Otherwise robots.txt may say “allow” while the CDN still blocks the crawler.

Anthropic warns that blocking its crawlers by IP address may not work reliably because it prevents them from reading robots.txt; it recommends robots.txt for opting out.

Other technical blockers

  • Client-side rendering. Bing advises against hiding critical content behind client-side rendering.
  • Meta directives. noindex removes a page from Google; Bing says NOARCHIVE prevents content from being used in Copilot responses.
  • Forms and logins. Content behind a login or form is not available to crawlers.

Sources

  1. OpenAI. Overview of OpenAI crawlers
  2. Claude Help Center. Does Anthropic crawl data from the web, and how can site owners block the crawler?
  3. Perplexity Docs. Perplexity Crawlers
  4. Google Search Central. Google’s common crawlers (Google-Extended)
  5. Google Search Central. AI features and your website
  6. Bing Webmaster Guidelines
  7. Cloudflare. Content Independence Day: no AI crawl without compensation! (1 July 2025)
  8. Cloudflare. Your site, your rules: new AI traffic options for all customers (1 July 2026)

FAQ

Should I block GPTBot?

It is your choice. OpenAI describes GPTBot as crawling content that may be used for training; appearing in ChatGPT search depends on OAI-SearchBot, a separate crawler.

Does blocking Google-Extended remove my site from Google or AI Overviews?

Google states that Google-Extended does not impact a site’s inclusion in Google Search and is not a ranking signal. AI Overviews use pages indexed by Googlebot.

Which crawler do I need for ChatGPT?

OAI-SearchBot. OpenAI says sites opted out of it are not shown in ChatGPT search answers.

Which crawler do I need for Claude?

Claude-SearchBot and Claude-User. Anthropic says disabling them may reduce your visibility in Claude’s search results.

Why do AI crawlers not reach my site even though robots.txt allows them?

Often because of a CDN or firewall: Cloudflare, for example, has blocked AI crawlers by default since July 2025. Check the bot settings there too.

Want AI to recommend your business?

We will check what ChatGPT, Gemini and Perplexity say about your niche today and show you where to start.

Discuss your project About the service WhatsApp

More articles