Guide

Bytespider: what it is and whether to block it

Bytespider is ByteDance's AI crawler with no published documentation, verification or IP ranges. Learn what it does and how to block it reliably.

By Sunny Patel · Updated 2026-08-23

Bytespider is ByteDance's web crawler. ByteDance owns TikTok. The crawl is widely understood to feed training data into ByteDance's large language models. Doubao is the main model connected to that training pipeline.

Site owners find this bot in their logs and search for an explanation. Bytespider requests pages the same way any crawler does. ByteDance never confirms why in public documentation. That gap between crawl activity and published explanation defines almost everything else on this page.

This page covers what the bot does, what independent reporting says about its behaviour and the specific steps that actually stop it. Robots.txt gets covered too. The reason robots.txt alone may not be enough for this particular crawler gets covered as well.

What Bytespider does

Bytespider crawls public pages much like Googlebot does. It follows links, downloads HTML and stores the content for later processing. ByteDance uses that stored content to train its large language models. The training relationship is well understood within the AI crawler community. ByteDance has never published a page confirming it. OpenAI, Anthropic and Perplexity each publish that kind of page for their own crawlers.

Its user-agent string carries one identifying token. That token is the word Bytespider. Matching that token is the only reliable way to spot this crawler in server logs. ByteDance provides no other identifier.

Training crawlers like Bytespider work differently to search-index crawlers. A search-index crawler builds the index an AI product answers questions from. A training crawler feeds a much larger, slower process that shapes a future model release. Blocking a training crawler does not remove a site from any live AI answer. It only opts that content out of future training runs. That protection depends entirely on the crawler honouring the block.

Why Bytespider is harder to manage than other AI crawlers

ByteDance publishes no crawler documentation for Bytespider. OpenAI, Anthropic and Perplexity each maintain a page describing their bot's purpose, behaviour and control options. ByteDance offers none of that for Bytespider.

No verification mechanism confirms a request carrying the Bytespider token actually comes from ByteDance's infrastructure. Anyone could spoof that user-agent string. Nothing on the receiving end would catch it. ByteDance has published no IP ranges either. Compare that to OpenAI, Anthropic and Perplexity. Each of them publishes IP ranges so site owners can verify traffic against a DNS or IP list.

That combination matters more than it first appears. A crawler with published documentation gives a site owner a policy to point at and a contact route if something goes wrong. A crawler with published IP ranges gives infrastructure teams a way to firewall it with confidence. Bytespider offers neither. Every decision about it rests on user-agent matching and third-party reporting instead of an official source.

What the reporting says about its crawl behaviour

Hosting providers and tech press reported extensively on Bytespider through 2024. Multiple reports described it as one of the highest-volume AI crawlers active on the web. Several of those reports documented cases of it ignoring robots.txt rules. ByteDance did not publish anything confirming or denying compliance with the standard.

Treat these as reported patterns rather than settled fact. No independent audit trail exists to confirm every case. The pattern across multiple independent sources is consistent enough that most security teams now treat Bytespider as unreliable by default.

Crawl volume adds a second problem beyond compliance. A high-volume bot that ignores disallow rules can pull real bandwidth and server load away from paying users. That operational cost sits alongside the training question. Neither one gets resolved soon. ByteDance has not published anything further on either point.

Does blocking Bytespider cost you AI search visibility

No. Blocking Bytespider carries no known visibility cost in Western AI search products. No ByteDance product cites Western sites back with referral traffic the way ChatGPT search or Perplexity do. That absence changes the entire calculation compared with other training crawlers.

Blocking GPTBot or ClaudeBot is a genuine tradeoff. You give up potential inclusion in future model training while search-index crawlers keep working normally. Bytespider offers no equivalent upside to weigh against that decision. Most sites lose nothing measurable by blocking it. That asymmetry explains why Bytespider shows up on more block lists than almost any other AI crawler.

GPTBot and ClaudeBot decisions require weighing that upside against training opt-out. Bytespider decisions do not. The absence of any Western citation product tied to ByteDance's crawl removes the entire upside half of the equation. What remains is closer to a default block than a judgement call.

How to block Bytespider

Robots.txt is the standard first step for any crawler. Add a dedicated block for Bytespider to your existing file.

User-agent: Bytespider
Disallow: /

That directive only works if Bytespider actually respects it. The reports above raise real doubt about that compliance. Treat robots.txt as a request rather than an enforcement mechanism for this specific bot.

Server-level and CDN-level blocking is the more reliable option for Bytespider.

• Match the token "Bytespider" in a firewall or WAF rule against the incoming user-agent header.
• Check whether your CDN has dedicated bot-management or AI-crawler controls.
• Cloudflare, for example, offers one-click AI crawler blocking that covers Bytespider without writing a custom rule.

Bytespider in the wider AI crawler picture

sitemap.digital's scanner checks robots.txt access across the major AI crawlers on any site you enter. 8 of the 43 sites scanned by sitemap.digital block at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot in robots.txt. That is 19%.

Bytespider sits outside that particular stat. It still belongs in the same audit process. AI vendors now run three kinds of agents against the open web. Training crawlers like GPTBot and ClaudeBot pull content into future models. Search-index crawlers like OAI-SearchBot and PerplexityBot build the index AI search products cite answers from. User-fetch agents like ChatGPT-User and Perplexity-User fetch a single page live when someone asks about it directly. Bytespider fits the training category. It carries none of the search-visibility upside that makes GPTBot and ClaudeBot a genuine tradeoff decision.

Confusing these categories is the most common mistake site owners make in robots.txt. Blanket-blocking every bot with the word AI in its name removes a site from citations it actually wants. Understanding which category a bot falls into comes before any blocking decision. Bytespider is no exception.

Separate guides cover GPTBot, ClaudeBot and PerplexityBot individually. A full AI crawlers list rounds out the picture for anyone auditing every bot at once.

Frequently asked questions