Guide

ClaudeBot: what it is and whether to block it

ClaudeBot is Anthropic's training crawler. Learn its user agent, how it differs from Claude-SearchBot and Claude-User, and how to block it safely.

By Sunny Patel · Updated 2026-08-23

ClaudeBot is Anthropic's web crawler. It gathers training data for Claude models.

The crawler identifies itself with a fixed user-agent token: ClaudeBot/1.0; +claudebot@anthropic.com. That token carries Anthropic's contact email directly inside the string. Site owners can reach the crawler operator without digging through separate documentation.

Confusion around ClaudeBot usually comes from three separate Anthropic agents sharing a similar name. Each agent does a different job and each responds to a different robots.txt rule. This guide separates the three, shows the exact user agent and explains when blocking ClaudeBot helps rather than hurts your visibility in Claude's answers.

sitemap.digital runs a free scan that checks robots.txt access for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. The same scan covers page-level AI readability, llms.txt and orphan pages.

ClaudeBot's user-agent string

ClaudeBot announces itself with one fixed token on every request it makes.

ClaudeBot/1.0; +claudebot@anthropic.com

That string does two jobs at once. It identifies the crawler and it gives site owners a direct contact address for questions or complaints.

Server logs, WAF rules and robots.txt directives can all match on this exact token. The core identifier to match on is ClaudeBot.

Matching the string is not the same as confirming the request came from Anthropic. That distinction matters enough to get its own section further down this page.

A missing token or a truncated string in your logs points to a different crawler entirely rather than ClaudeBot. Wildcard rules that only match "Claude" catch ClaudeBot, Claude-SearchBot and Claude-User at once. Narrower rules that match the full ClaudeBot token catch training traffic only.

What ClaudeBot does with your content

ClaudeBot crawls pages to build training data for Claude models.

Anthropic documents that ClaudeBot respects robots.txt directives. A correctly formatted Disallow rule for ClaudeBot should stop the crawler from fetching those paths.

Training crawlers work differently from the agents behind Claude's live search answers. ClaudeBot reads your content once to help shape a future model. It does not decide whether your pages show up when someone asks Claude a question today.

That distinction is the single most common source of misconfigured robots.txt files. Site owners block ClaudeBot expecting to protect their search visibility and instead only affect model training.

The training relationship is one-directional. ClaudeBot reads your page during a crawl pass and that content may later inform a future model version. It does not create an ongoing live connection to your site the way Claude-User does.

ClaudeBot, Claude-SearchBot and Claude-User

Anthropic runs three separate agents under the Claude name. Each one does a different job.

Mixing them up is the easiest way to lose Claude citations by accident.

Blocking ClaudeBot alone leaves Claude-SearchBot and Claude-User untouched. Your content keeps appearing in Claude's search answers even after you opt out of training.

Blocking all three at once removes your content from training data, from Claude's search index and from live fetches when someone asks Claude about your page directly.

Treat the three names as three separate switches rather than one general Claude switch in your robots.txt.

Claude-Web is retired

Claude-Web no longer crawls or fetches anything.

Claude-SearchBot and Claude-User replaced it entirely. They took over search indexing and live fetching respectively.

A robots.txt rule naming claude-web controls nothing current.

Many old block lists still carry that line. Copying an old block list without checking it against current agent names achieves nothing beyond wasted lines in your robots.txt.

Auditing your current robots.txt for a lingering claude-web line takes a few minutes. Removing it clears a rule that no longer does anything.

Should you block ClaudeBot

Blocking ClaudeBot opts your content out of Claude model training. It does nothing to your presence in Claude's search answers.

Blocking Claude-SearchBot removes your pages from the index Claude's search answers cite from. Blocking Claude-User stops Claude from fetching that page live when a person asks about it directly.

8 of the 43 sites scanned by sitemap.digital block at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot in robots.txt. That is 19%. The scan reads robots.txt permissions rather than observed crawler traffic. That figure describes what site owners have configured rather than what actually gets crawled.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

This configuration blocks training only. Search visibility and live fetches stay intact.

Blocking all three agents blocks everything: training, search and live fetches alike.

Decide per agent rather than blanket-blocking anything with Claude in the name.

A rule covering only the word Claude in the agent name catches all three agents by accident. Naming each agent explicitly avoids that mistake. The agent name in a rule decides its effect rather than the intent behind writing it.

Verifying ClaudeBot in your server logs

User-agent strings can be spoofed by any client. A UA match alone does not prove Anthropic ran the request.

Treat raw log counts of ClaudeBot hits as an upper bound rather than a confirmed figure.

Treat a UA match as a starting point for investigation rather than final proof of origin.

Pair a UA check with a wider review of the requesting behaviour before drawing conclusions about who is actually crawling your site.

The site's full AI crawlers list covers matching rules for every major agent. Dedicated pages for GPTBot, PerplexityBot and Bytespider go deeper on each one.

Frequently asked questions