Guide

AI crawlers: every major bot and what it does

Every major AI crawler explained by vendor and job: GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot and Google-Extended. Includes a robots.txt example.

By Sunny Patel · Updated 2026-08-23

AI crawlers are automated bots that vendors like OpenAI, Anthropic, Perplexity, Google and ByteDance send to your site for three distinct jobs. Training crawlers pull content to build future models. Search-index crawlers build the index that AI search products cite answers from. User-fetch agents grab a single page the moment someone asks about it in a chat. Confusing these three roles causes most robots.txt mistakes on this list.

This page lists every major AI crawler active in 2026. That includes training bots, search-index bots and user-fetch agents from OpenAI, Anthropic, Perplexity, ByteDance, Common Crawl, Meta and Google. It also covers Google-Extended, a control token rather than a crawler. A working robots.txt example for a selective blocking policy sits near the end.

Three jobs, three types of AI bot

AI vendors run three kinds of agents, each with a different effect on your site. Training crawlers bulk-crawl content to build future model weights. GPTBot and ClaudeBot fall into this category. Blocking a training crawler opts your content out of that vendor's future model training. It does not touch your visibility in search or chat answers today.

Search-index crawlers build the index AI search products query when a person asks a question. OAI-SearchBot, Claude-SearchBot and PerplexityBot do this job. Blocking one of these removes your pages from that product's cited answers.

User-fetch agents work differently again. ChatGPT-User, Claude-User and Perplexity-User fetch a single page live, triggered by a person's specific request inside a chat session. Blocking a user-fetch agent fails that specific live lookup for your domain.

Most sites only need to make one decision per vendor. Decide whether you want training access, search citations, both or neither before writing any robots.txt rule.

Every major AI crawler and what it does

The table below covers every major AI crawler by vendor, agent name and job. The final column shows what you lose by blocking it.

ClaudeBot, GPTBot, Bytespider and PerplexityBot each have a dedicated deep-dive page covering full behaviour and blocking guidance.

8 of the 43 sites scanned by sitemap.digital block at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot in robots.txt. That is 19%. The scanner reads robots.txt permissions rather than observed crawler traffic.

Google-Extended is not a crawler

Google-Extended is a robots.txt control token rather than a crawler. Disallowing Google-Extended opts your content out of training for Google's Gemini models. Googlebot still crawls and indexes your site as normal.

AI Overviews visibility follows standard Google Search eligibility. Googlebot governs that eligibility rather than Google-Extended. Blocking Googlebot to avoid AI Overviews also removes you from normal Google search results entirely.

How to block AI crawlers in robots.txt

Robots.txt rules match by product-token prefix under a named User-agent line. A blanket User-agent: * group does not apply to a bot that has its own named group elsewhere in the file. Each vendor's agents need their own explicit rule if you want different treatment for training versus search versus user-fetch.

The example below blocks GPTBot for training while leaving OAI-SearchBot free to build ChatGPT's search index.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Repeat the same pattern per vendor. Block the training agent. Allow the search-index and user-fetch agents to keep citations without adding to training data.

Publish the file at the domain root rather than inside a subdirectory. Crawlers only check /robots.txt on the root domain.

Verify before you trust a user-agent string

User-agent strings in server logs can be spoofed by anyone. A line reading GPTBot in your logs is not proof OpenAI's crawler made that request.

Vendor IP lists are the only reliable verification method. OpenAI and Perplexity both publish IP ranges for their crawlers. Cross-check any suspicious hit against the published list before you trust the user-agent string alone.

Scrapers unrelated to AI vendors sometimes borrow AI crawler names to blend into log noise or bypass simple user-agent blocks. Treat unverified hits as unknown traffic rather than confirmed AI crawler activity.

Common robots.txt mistakes

Most robots.txt AI-blocking rules fail for one of a handful of repeat reasons. Each one is easy to fix once you see it.

• Naming claude-web instead of the current agents. Anthropic retired Claude-Web. Claude-SearchBot and Claude-User replaced it. Old rules naming claude-web control nothing current.
• Blocking everything with 'AI' in the name. A blanket approach blocks training, search-index and user-fetch agents together and kills citations along with training access.
• Assuming a wildcard User-agent: * rule covers every bot. Rules match by product-token prefix. A named group for a specific bot overrides the wildcard group for that bot.
• Trusting user-agent strings without IP verification. Spoofed hits produce misleading crawl logs.
• Blocking Googlebot to dodge AI Overviews. That also removes the site from normal Google search.

Frequently asked questions