Bytespider (ByteDance): what it is and how to allow or block it
Bytespider is ByteDance's crawler. We found no official ByteDance documentation page for it. Its robots.txt token is Bytespider.
Facts
- Operator
- ByteDance
- Category
- AI training crawler
- robots.txt token
Bytespider
- Honors robots.txt?
- Reports differ. Independent reports disagree on whether the crawler always respects robots.txt.
- User-agent example
- Contains
bytespider(the operator documents no full example string)
- Official source
- No official ByteDance documentation found
- Compared with the source
What it is used for
Third-party crawler directories describe it as gathering content for ByteDance products such as Toutiao search and AI model training; ByteDance itself does not clearly document the use. Collects public pages that may be used to train AI models. Blocking it is a statement about training, not about being found.
What happens if you block it
A User-agent: Bytespider rule is the commonly published opt-out.
Because there is no official page and no published IP list, a request that says Bytespider cannot be verified; the user-agent string is only a claim.
How to allow or block Bytespider
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: Bytespider Allow: /
User-agent: Bytespider Disallow: /
Does your site allow Bytespider today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
No request identifying as Bytespider has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
Other AI training crawlers
- GPTBot OpenAIOpenAI's crawler for content that may be used to train its generative AI models.
- ClaudeBot AnthropicAnthropic's crawler that collects web content which could contribute to training its generative AI models.
- CCBot Common CrawlThe crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data.
- meta-externalagent MetaMeta's crawler that gathers web content to train AI models or to improve products by indexing.
All AI Training Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot