Guide

PerplexityBot: what it is and how to control it

Learn what PerplexityBot is, how to verify it with IP ranges, and how to block it in robots.txt without losing Perplexity's citation traffic.

By Sunny Patel · Updated 2026-08-23

PerplexityBot is Perplexity's search-index crawler. It builds the index Perplexity cites answers from. Perplexity itself is an AI answer engine rather than a traditional search engine. It answers questions directly and shows the sources behind each answer.

Site owners search for PerplexityBot for one of two reasons. The crawler shows up in server logs and needs identifying. A decision also needs making about whether to allow it at all. Both reasons come down to the same question. What happens to a site's presence in Perplexity's answers if PerplexityBot gets blocked?

This page covers PerplexityBot's user agent string, verification, and robots.txt control. It also covers Perplexity-User and the stealth crawler dispute with Cloudflare.

None of this requires guesswork. Perplexity's crawler behaviour is documented and controllable through standard robots.txt syntax.

PerplexityBot and Perplexity-User: two different agents

Perplexity runs two separate agents. PerplexityBot crawls in bulk and builds Perplexity's search index. Perplexity-User fetches a single page live when a person asks Perplexity about it.

The distinction matters for robots.txt. Perplexity's own documentation states Perplexity-User may not honour robots.txt. The fetch is user-initiated rather than a bulk crawl. Blocking PerplexityBot in robots.txt does not necessarily stop Perplexity-User from fetching a page on demand.

A robots.txt rule targeting PerplexityBot leaves Perplexity-User unaffected. Perplexity-User's behaviour follows a person's request rather than a fixed crawl schedule. Treat the two agents as separate control decisions rather than one.

PerplexityBot's user agent string

PerplexityBot identifies itself with a specific token in its user agent header. Check server logs for this exact string to flag candidate PerplexityBot hits. Verify each candidate against Perplexity's published IP ranges before trusting it.

PerplexityBot/1.0; +https://perplexity.ai/perplexitybot

Match the full token rather than a fragment. Two different bots can both include the word Perplexity in a longer string.

A generic search for the word Perplexity in log files can return unrelated hits from browsers, extensions, or other tools that mention the brand. Filter for the exact token above to isolate genuine PerplexityBot requests.

Verifying PerplexityBot beyond the user agent

A user agent string alone proves nothing. Anyone can set a request's user agent to PerplexityBot/1.0 and spoof a crawl.

Perplexity publishes IP ranges for verification. Cross-check the requesting IP against Perplexity's published ranges before trusting a user agent claim alone. Log analysis tools and CDN bot-management products both support this kind of IP-range verification.

IP-range verification catches the gap a user agent check leaves open. A request claiming to be PerplexityBot that originates outside Perplexity's published ranges is not PerplexityBot. Treat it as an unverified crawler regardless of what header it presents.

Combine both checks for a reliable verdict. A hit that matches the user agent token and falls inside a published IP range is genuine PerplexityBot traffic. A hit that fails either check should not be trusted as PerplexityBot regardless of what it claims to be.

Blocking PerplexityBot in robots.txt

Disallow PerplexityBot entirely with two lines in robots.txt.

No partial version exists. A site cannot stay in Perplexity's answers while opting the underlying crawl out.

User-agent: PerplexityBot
Disallow: /

This removes the site from Perplexity's index. Pages already cited in Perplexity answers drop out over time.

Perplexity sends real referral traffic through citations. This block therefore carries a visible cost unlike blocking a pure training bot.

8 of the 43 sites scanned by sitemap.digital block at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot in robots.txt. That is 19%.

Run a scan before deciding. Check which AI crawlers are already blocked on the site and which are still allowed.

A scan shows the current state of every major AI crawler in one pass rather than a manual robots.txt read.

The stealth crawler dispute with Cloudflare

Cloudflare published research in August 2025 alleging Perplexity used undeclared stealth crawlers. Cloudflare said these crawlers used generic browser user agents to fetch content from sites that had already blocked Perplexity's declared bots.

Perplexity disputes Cloudflare's findings. Perplexity says the traffic in question was user-driven fetching rather than undeclared crawling.

Robots.txt only governs declared agents. CDN-level bot management is the stronger enforcement layer for a site owner who wants a hard block regardless of what user agent a request claims.

The dispute matters beyond Perplexity. It signals that a declared user agent and a robots.txt rule are not a guaranteed technical barrier. Bot management at the CDN or WAF layer inspects behaviour and IP reputation rather than trusting a header.

Treat robots.txt as policy rather than enforcement. A hard block against any AI crawler, declared or undeclared, requires inspecting requests at the network layer rather than trusting a text file alone.

PerplexityBot compared to other AI crawlers

AI vendors run three kinds of agents. Training crawlers bulk-crawl to train models. Search-index crawlers build the index an AI answer engine cites answers from. User-fetch agents retrieve a single page live when someone asks about it.

PerplexityBot is a search-index crawler. Perplexity does not run a separate training crawler the way OpenAI does with GPTBot and OAI-SearchBot. The practical decision for PerplexityBot is mostly binary: index in or index out.

Separate guides cover GPTBot, ClaudeBot, and Bytespider individually. A full list of every AI crawler active on the web today covers the rest.

Deciding whether to allow PerplexityBot works the same way as deciding on any search-index crawler. Weigh the citation traffic a site would lose against the content it would rather keep out of a public AI answer.

A scan against sitemap.digital's checklist shows exactly where PerplexityBot stands on a site right now. The same scan checks other AI visibility signals like llms.txt and page-level readability.

Frequently asked questions