Guide

OAI-SearchBot explained: the crawler behind ChatGPT search

OAI-SearchBot builds the index ChatGPT search cites from. See its user agent string, how it differs from GPTBot, and how to check access.

By Sunny Patel · Updated 2026-09-20

OAI-SearchBot is the OpenAI crawler that builds the search index ChatGPT search draws its citations from. A page missing from that index cannot be cited in a ChatGPT search answer, regardless of how well it ranks anywhere else. It is one of three separate OpenAI agents, and confusing it with either of the other two is the single most common mistake site owners make when deciding what to allow. This guide covers what OAI-SearchBot specifically does, the exact robots.txt rule that keeps it allowed while other OpenAI agents stay blocked, and how to confirm your own site's current access.

OAI-SearchBot versus GPTBot versus ChatGPT-User

GPTBot gathers content that may be used to train future OpenAI models. That crawl runs on an ongoing schedule and has nothing to do with what ChatGPT can cite today. OAI-SearchBot does a different job entirely: it crawls and indexes pages specifically so ChatGPT search can surface and link to them in an answer. A page can be fully blocked from GPTBot and still show up in ChatGPT search citations, as long as OAI-SearchBot keeps separate access. ChatGPT-User is a third agent again, one that fetches a single page live at the moment a person asks ChatGPT to open or browse a specific link during a conversation, distinct from both the training crawl and the standing search index.

Site owners who block "everything with GPT in the name" in one robots.txt block usually mean to opt out of training. They end up losing ChatGPT search citations at the same time, without realising the two were ever separate settings.

The user agent string

OAI-SearchBot identifies itself with the token OAI-SearchBot/1.0; +https://openai.com/searchbot. The version number attached to the token can change, so a robots.txt rule written against the full string including that version number can silently stop matching after an update. Write the rule against the product token, OAI-SearchBot, which robots.txt matches by prefix and which keeps working across every future version.

OAI-SearchBot/1.0; +https://openai.com/searchbot

How to allow or block it in robots.txt

OAI-SearchBot needs its own named user-agent group, separate from any rule written for GPTBot or ChatGPT-User. A named group for one agent does not extend to the others, and a general User-agent: * block only applies when no more specific named group exists for that agent.

To keep ChatGPT search citations while opting out of training, allow OAI-SearchBot explicitly and disallow GPTBot in its own separate block:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

To remove your pages from ChatGPT search citations entirely, disallow OAI-SearchBot on its own:

User-agent: OAI-SearchBot
Disallow: /

OpenAI's own documentation confirms that its crawlers, OAI-SearchBot included, respect robots.txt, so a correctly scoped disallow rule under its named block is the standard way to opt out.

Verifying it in your logs

A log line reading OAI-SearchBot proves nothing on its own, for the same reason a GPTBot line does not: any client can set that string on a request OpenAI never sent. Check the source IP against OpenAI's published IP ranges at developers.openai.com/api/docs/bots, the authoritative source for the current list, instead of a copy that may be out of date. The GPTBot guide covers this verification step for all three OpenAI agents in one place.

What counts as a genuine allow

An allow rule for OAI-SearchBot only removes a robots.txt barrier. It says nothing about whether the crawl actually found the page in the first place. A page that sits behind no disallow rule at all can still miss OAI-SearchBot's crawl entirely if nothing on the site links to it, since a crawler without a link cannot reach a page regardless of what robots.txt permits. Treat the allow rule as a necessary condition, not a guarantee, and confirm separately that the page carrying the content you want cited actually sits inside your normal internal link graph.

Why blocking it removes you from ChatGPT search answers

ChatGPT search cites from the index OAI-SearchBot builds. A page that agent cannot reach never enters that index, and a page never in the index cannot be surfaced or linked to in a ChatGPT search result, no matter how strong that page's content or backlink profile is elsewhere. This is a direct mechanical consequence, not a probability. Sites frequently block OAI-SearchBot by accident while trying to opt out of training, because a disallow rule aimed at "OpenAI" or "GPT" in a shared block catches every OpenAI agent at once instead of the one actually intended. The full AI crawlers list covers the same training-versus-search distinction across every major vendor, not just OpenAI, since the mistake repeats across Anthropic and Perplexity's crawlers in exactly the same shape.

What OAI-SearchBot does not do

OAI-SearchBot does not gather content for model training. That job belongs to GPTBot alone. It also does not perform a live, single-page fetch triggered by a specific chat message; that is ChatGPT-User's job. OAI-SearchBot runs a standing crawl aimed at keeping a search index current, closer in behaviour to Bingbot or Googlebot than to either of the other two OpenAI agents. A site that wants ChatGPT search citations without contributing to model training keeps this agent allowed while disallowing GPTBot in a separate named block, exactly as shown above.

This distinction matters most for a site that earns real traffic from being cited by name inside a ChatGPT search answer, since access for that specific crawler is what the citation depends on. A site can score well on every other AI readiness check, keep GPTBot fully blocked for privacy reasons, and still show up in ChatGPT search results, provided OAI-SearchBot itself was never swept up in the same block.

Checking your own site

Run the AI crawler access checker to see whether OAI-SearchBot currently has access to your site, alongside GPTBot, ChatGPT-User, and the other major AI and search crawlers, in one pass over your robots.txt. The tool reports whether each agent is allowed, blocked, partially blocked, or has no rule at all, and whether an llms.txt file exists at the root. It reads permission rules only. It does not confirm a real crawl happened, since no operator on that list publishes public per-site traffic data.

The full site audit goes beyond the robots.txt permission check and looks at whether OAI-SearchBot could actually read and follow the content once it arrives, including click depth, orphan pages, and broken internal links that a permission rule alone cannot show.

Frequently asked questions