Guide

GPTBot, OAI-SearchBot and ChatGPT-User explained

GPTBot, OAI-SearchBot and ChatGPT-User are three separate OpenAI crawlers with different jobs. See the UA tokens and the robots.txt rules that control each one.

By Sunny Patel · Updated 2026-08-23

OpenAI runs three separate crawlers under three different names. GPTBot gathers training data for OpenAI's models. OAI-SearchBot builds the index that ChatGPT search cites answers from. ChatGPT-User fetches a single page live when someone asks ChatGPT about it. Each agent carries its own user-agent token and its own robots.txt rule.

Confusing the three is the single most common mistake site owners make here. A site owner blocks GPTBot expecting to disappear from ChatGPT entirely. GPTBot only controls training data collection. OAI-SearchBot controls whether ChatGPT search can cite the page at all.

Sites that disallow all three in robots.txt lose citations without realising why. Training opt-out and search visibility are separate decisions. Getting this wrong costs real citation traffic for no privacy benefit.

Three OpenAI crawlers with three different jobs

GPTBot crawls the web to gather training data for OpenAI's models. Content it collects can shape how future model versions respond to questions. The crawl runs on an ongoing basis and touches most public pages that allow it. A site with no GPTBot rule at all is fully open to this crawl by default.

OAI-SearchBot builds the search index ChatGPT search draws on. That index is the source for citations inside ChatGPT search answers. A page missing from this index cannot be cited. Google ranking has no bearing on this particular citation path. A page can rank first on Google and still be invisible to ChatGPT search.

ChatGPT-User fetches a page live and in real time. It triggers when a person is actively chatting with ChatGPT and asks about a specific page or site. The fetch happens once at the moment of the question rather than on a crawl schedule. This behaviour looks like a single human visit in server logs rather than a crawler sweep.

How to identify each agent in your logs

Each crawler announces itself with a distinct user-agent token. GPTBot uses the token "GPTBot" with a reference URL of openai.com/gptbot. OAI-SearchBot identifies as "OAI-SearchBot/1.0; +https://openai.com/searchbot". ChatGPT-User carries the token "ChatGPT-User" with a reference URL of openai.com/bot.

Version numbers inside these strings change over time. OpenAI has updated the OAI-SearchBot version number before without warning. The product token is the stable part robots.txt actually matches on.

Match rules against the token rather than the full string. A rule written for "GPTBot" catches every version of that crawler automatically. A rule written against the exact full string with a version number breaks the next time OpenAI ships an update.

GPTBot (openai.com/gptbot)
OAI-SearchBot/1.0; +https://openai.com/searchbot
ChatGPT-User (openai.com/bot)

Verify before you trust the log line

A user-agent string proves nothing on its own. Any client can set its user-agent to "GPTBot" and send requests that were never sent by OpenAI. Scrapers do this constantly to slip past simple bot-detection rules.

OpenAI publishes IP address lists for its agents. Checking the source IP against those published lists is the only way to confirm a request genuinely came from GPTBot, OAI-SearchBot or ChatGPT-User. Log analysis that skips this step ends up measuring spoofed traffic alongside real traffic.

GPTBot respects robots.txt

OpenAI documents that GPTBot respects robots.txt. A disallow rule for the GPTBot token stops the crawler from fetching those paths. This makes robots.txt the correct control point for training opt-out rather than a meta tag or a firewall rule.

Robots.txt matches by product-token prefix. A named group for GPTBot overrides the general User-agent: * block for that bot specifically. A wildcard block alone will not stop GPTBot if a GPTBot-specific allow rule sits below it in the file. Order and grouping inside the file matter as much as the individual rule.

Robots.txt example: block training, keep search visibility

Most sites want to opt out of training while staying visible in ChatGPT search and live fetches. That takes three separate rules rather than one blanket block. Each bot needs its own named group in robots.txt.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

This blocks GPTBot from crawling any path on the site. OAI-SearchBot and ChatGPT-User keep full access. Citations and live fetches continue as normal. Test the rule with a robots.txt checker after publishing it. A syntax error in one named group can silently affect the others.

The expensive mistake: blanket AI blocks

Many sites copy a blanket AI-block list from a forum post or plugin default. That list disallows GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and everything else with "AI" in the name. The list gets pasted in once during a security review and never revisited.

The site then wonders why ChatGPT never cites it. Training opt-out and search visibility are separate decisions. Blocking the wrong bot silently kills citations while doing nothing for training privacy.

8 of the 43 sites scanned by sitemap.digital block at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot in robots.txt. That is 19%.

Guides for ClaudeBot, PerplexityBot and Bytespider cover the same trap for other vendors. A full AI crawlers list rounds up every agent worth checking. Reviewing robots.txt against that list once a quarter catches drift before it costs citations.

Auditing GPTBot access on your own site

Checking current rules takes a few minutes. Fetch the live robots.txt file and search it for the three tokens covered on this page.

• Search robots.txt for "GPTBot", "OAI-SearchBot" and "ChatGPT-User" as separate named groups.
• Confirm which paths each group disallows rather than assuming one wildcard rule covers all three.
• Check server logs for the exact tokens. Verify hits against OpenAI's published IP ranges.
• Decide training opt-out and search visibility as two separate choices rather than one setting.
• Re-check the file after any theme, plugin or CDN change. Those tools rewrite robots.txt without warning.
• Run a fresh scan on a schedule rather than once at launch. Rules drift as plugins update and hosting changes.

Frequently asked questions