JW2D OÜ

How to appear in Claude answers: Anthropic’s crawlers and web search

Yakiv Romanchuk, founder of JW2D · Published

Short answer

Claude searches the web in real time and, according to Anthropic, every response based on web search includes citations. For your site to be among them, Anthropic’s crawlers must be able to reach it: Claude-SearchBot indexes content for search, and blocking it may reduce your visibility; Claude-User fetches pages when a user asks; ClaudeBot collects content for model training. They are controlled separately in robots.txt, so you can opt out of training and still stay visible in search. Claude is available in Estonia.

Anthropic’s three crawlers

Agent What it does (according to Anthropic) If you block it
Claude-SearchBot Navigates the web to improve search result quality for users Your content is not indexed for search, which may reduce your site’s visibility and accuracy in user search results
Claude-User Supports Claude users: retrieves pages when a user asks a question Claude cannot retrieve your content in response to a user query
ClaudeBot Collects web content that could potentially contribute to model training Your site’s future materials are excluded from training datasets

To appear in Claude answers, allow at least Claude-SearchBot and Claude-User. ClaudeBot is a separate decision about training; blocking it does not block the other two.

robots.txt, crawl rate and firewalls

  • Anthropic’s bots honor robots.txt directives. The rule must be in the robots.txt of every subdomain.
  • To limit load, Anthropic supports the non-standard Crawl-delay directive.
  • Anthropic does not recommend blocking its IP addresses: then the bot cannot read robots.txt, and the opt-out may not work reliably. Anthropic publishes the list of its crawler IP addresses.
  • A CDN or firewall can block crawlers even if robots.txt allows them. For example, on 1 July 2025 Cloudflare announced that it was changing its default to block AI crawlers — see which AI crawlers to allow.

Example of a robots.txt that keeps a site visible in Claude search but excludes it from training:

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

On jw2d.com all three are allowed: we want our texts to be known to AI models as well.

How Claude web search works

  • Anthropic writes that web search gives Claude access to real-time data and that every response includes citations, so users can check the sources.
  • In the standard chat interface, the user turns web search on; in the newer Claude interface, Claude searches the web when it helps.
  • On Team and Enterprise plans, an Owner must first enable web search in organization settings.
  • Free accounts can use search within usage limits.
  • Anthropic’s help article does not name the provider of general web search results; it only says image results are powered by Bing.

Does Claude work in Estonia

Yes. Estonia is on Anthropic’s list of supported regions for its apps, including Claude.ai.

What to do

  1. Allow Claude-SearchBot and Claude-User in robots.txt and check the CDN or firewall settings.
  2. Put the facts in the text. Services, city, prices, contacts and working hours in plain HTML text, the same on every page (our practice).
  3. Answer the question at the start. Claude cites specific sources, so a page that answers the question in the first sentences is easier to quote (our practice).
  4. Add sources and numbers. In the GEO research, quotations, statistics and sources increased visibility in AI answers. Claude was not among the tested systems, so this is a general recommendation.
  5. Keep the site indexed in Google and Bing too: it helps in all AI search systems — see how AI chooses sources.

How to check whether Claude cites you

We have not found an official Claude report for site owners. Ask Claude your customers’ questions with web search on, in their languages, and record whether your site is among the citations and who is recommended instead. In the server logs you can see visits from Claude-SearchBot and Claude-User by user agent (our method).

Sources

  1. Claude Help Center. Does Anthropic crawl data from the web, and how can site owners block the crawler?
  2. Claude Help Center. Enabling and using web search
  3. Anthropic. Supported Regions Policy
  4. Cloudflare. Content Independence Day: no AI crawl without compensation
  5. Aggarwal P. et al. GEO: Generative Engine Optimization. arXiv:2311.09735, KDD 2024

FAQ

Which Claude crawler is needed to appear in Claude answers?

Claude-SearchBot, which indexes content for search, and Claude-User, which fetches pages when a user asks. Anthropic writes that blocking Claude-SearchBot may reduce your visibility in search results.

Can I opt out of Claude training but stay in search?

Yes. Training is controlled by ClaudeBot, separately from Claude-SearchBot and Claude-User. Disallow ClaudeBot in robots.txt and allow the other two.

Does Claude show sources?

Anthropic writes that every response based on web search includes citations.

Is Claude available in Estonia?

Yes. Estonia is on Anthropic’s list of supported regions.

Should I block Anthropic by IP address?

Anthropic does not recommend it: the bot then cannot read robots.txt, and the opt-out may not work reliably. Use robots.txt.

Want AI to recommend your business?

We will check what ChatGPT, Gemini and Perplexity say about your niche today and show you where to start.

Discuss your project About the service WhatsApp

More articles