AI crawler access checker: is your site blocking ChatGPT, Claude, and Perplexity?
By Sunny Patel · Updated 2026-07-15
Paste a URL below and this tool reads that site’s robots.txt in one pass. It reports whether each of the 8 major AI and search crawlers is allowed in, blocked, or has no rule at all. It also reports whether an llms.txt file exists at the root. This is the check most site owners need when they ask whether their site blocks ChatGPT. It does not crawl the whole site, so results return in seconds.
Results appear here once you check a URL above.
What each of the 8 crawlers does
“AI crawler” is not one thing. Some of these user agents fetch content to train a future model. Some fetch a single page live in response to a user request. Two are the same general search crawlers that have existed for years, now doubling as the discovery layer behind AI Overviews and Copilot. Blocking the wrong one removes you from search results you actually want to keep.
| Crawler | Operator | What it does |
|---|---|---|
| GPTBot | OpenAI | Crawls pages that may be used to train future OpenAI models such as GPT and ChatGPT. |
| ChatGPT-User | OpenAI | Fetches a page live when a ChatGPT user asks it to browse or open a specific link. |
| ClaudeBot | Anthropic | Crawls pages that may be used to train future Claude models. |
| Claude-Web | Anthropic | Fetches a page live when a user asks Claude to look at a specific link. |
| PerplexityBot | Perplexity | Indexes pages so the Perplexity answer engine can surface and link to them in results. |
| Google-Extended | Controls whether a site’s content can be used to train Gemini and ground AI Overviews. It rides on Googlebot’s crawl rather than running a separate one. | |
| Bingbot | Microsoft | Microsoft’s general search crawler. The same crawl also sources citations shown in Copilot. |
| Googlebot | Google’s general search crawler. A page must be reachable by Googlebot to be indexed in Search or considered for AI Overviews. |
OpenAI’s own documentation confirms GPTBot and ChatGPT-User check robots.txt and lists their stated purpose in full at developers.openai.com/api/docs/bots. No operator in the table above publishes per-site traffic logs. A robots.txt rule is a permission signal, not proof of a real visit.
Why blocking a bot changes what an AI system knows about you
A disallow rule under a bot’s user-agent block is a request. Every crawler on this list honours that request voluntarily, according to its own published documentation. Block a training crawler such as GPTBot or ClaudeBot, and that operator’s stated policy is to leave your content out of future training data.
Block a citation or search crawler such as PerplexityBot, Bingbot, or Googlebot instead, and the effect is closer to disappearing from that engine’s results and answers. Those three crawlers depend on an active crawl to have anything to show or cite. Many sites block training crawlers by accident while trying to protect content. The rule usually sits in a shared robots.txt block that also touches a search crawler they still want.
How to unblock a bot in robots.txt
robots.txt lives at the root of a domain, for example https://example.com/robots.txt. Each bot reads the most specific matching user-agent block it can find. It falls back to a wildcard * block only when no named block exists. Add or edit a block like this to unblock GPTBot specifically:
User-agent: GPTBot
Allow: /A bot-specific Allow: / block always overrides an earlier wildcard disallow for that named bot, since a matching named block takes priority over the wildcard fallback. Remove the bot from any existing Disallow: / line under its own block rather than deleting the whole file. Other rules in the same file, such as your sitemap reference, usually still apply.
Frequently asked questions
Is my site blocking ChatGPT if GPTBot shows blocked here?
It means your robots.txt currently disallows GPTBot from crawling. OpenAI states it honours robots.txt, so a disallow rule should stop that specific crawler from fetching your pages for training. It does not tell you whether ChatGPT mentions your brand from other sources it already holds.
Does this tool show real traffic from these bots?
No. It reads the permission rule in your robots.txt, the same file any crawler is expected to check before requesting a page. It does not read server logs and cannot confirm a bot actually visited, because none of these operators publish public per-site traffic data.
What does "none" mean in the results table?
It means your robots.txt has no rule naming that bot and no wildcard rule blocking everyone, so the bot is free to crawl by default. This is the most common result for sites that have never edited robots.txt.
Does blocking these bots help or hurt my site?
That depends on your goal. Blocking a training crawler such as GPTBot or ClaudeBot keeps your content out of that company’s future model training data. Blocking a citation or search crawler such as PerplexityBot, Bingbot, or Googlebot instead removes you from that engine’s results and answers entirely, which is rarely the intended outcome.
How do I unblock a bot once I see it is blocked here?
Edit robots.txt at your site root and remove or narrow the disallow rule under that bot’s user-agent block, then rescan to confirm the change took effect. The how-to-unblock section on this page has the exact syntax.
This check covers robots.txt access only. See how AI crawlers read your whole site, page by page.
Run the full scan