Twitterbot (X Corp): what it is and how to allow or block it
Updated · checked against the operator's documentation on
Twitterbot is the fetcher X uses to build preview cards for links in posts. Its robots.txt token is Twitterbot.
Facts
- Operator
- X Corp
- Category
- Social preview fetcher
- robots.txt token
Twitterbot
- Honors robots.txt?
- Reported yes (older X docs). According to X's older documentation, Twitterbot implements Google's robots.txt specification.
- User-agent example
Twitterbot/1.0
- User agent we logged
Twitterbot/1.0Most common variant in our own logs, 4 requests, 19 to 25 Oct 2026. Static snapshot, not the live counter below. A logged string is a claim: anyone can send it.
- Official source
- X: Troubleshooting Cards (not readable on 2026-10-25)
- Compared with the source
What it is used for
Reads the card markup of a linked page so X can show a preview card. According to X's older developer documentation, the card is cached for about seven days.
Fetches a page when someone shares its link, to build the preview card (title, description, image). It acts for the person who shared the link.
What happens if you block it
Per X's older documentation, links to a blocked page get no preview card on X, and a blocked image URL means no thumbnail. We could not confirm this against X's current text.
X's current developer page returned an error when we fetched it on 2026-10-25, so these statements come from archived documentation and have not been confirmed against the current text.
Is it really Twitterbot?
Anyone can send Twitterbot as a user agent, and X Corp publishes no IP list or reverse-DNS method that we could find, so a log line cannot prove a request is real. Check whether it fetched /robots.txt and whether the address belongs to the operator's network.
- We did not find an IP list or a reverse-DNS method in the official source we compared on 2026-10-25, so the user agent alone cannot show that a request is genuine. Check the operator's page for updates.
- Paste any user agent from your logs into the user-agent lookup, or follow the scripts in Fake Googlebots and PHP scanners.
How to allow or block Twitterbot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: Twitterbot Allow: /
User-agent: Twitterbot Disallow: /
Does your site allow Twitterbot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
Live counter, last 30 days: no request identifying as Twitterbot recorded yet. The counter started after the week below, so it does not cover it. Earlier log snapshot (19 to 25 Oct 2026): 4 requests identified as Twitterbot.
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Questions about Twitterbot
What is Twitterbot?
The fetcher X uses to build preview cards for links in posts. Reads the card markup of a linked page so X can show a preview card. According to X's older developer documentation, the card is cached for about seven days.
Is Twitterbot safe and legitimate?
The real Twitterbot is a social preview fetcher run by X Corp. A user-agent string is only a claim, though, so anyone can send it. We did not find an IP list or reverse-DNS method in the official source we compared on 2026-10-25, so the user agent alone cannot prove a request is real.
Does Twitterbot obey robots.txt?
Reported yes (older X docs). According to X's older documentation, Twitterbot implements Google's robots.txt specification.
How do I block Twitterbot?
Robots.txt may not stop it. According to X's older documentation, Twitterbot implements Google's robots.txt specification. If you add a group anyway, use User-agent: Twitterbot followed by Disallow: /.
Will blocking Twitterbot hurt my Google rankings or AI visibility?
Not your rankings, but shared links can lose their preview card. Per X's older documentation, links to a blocked page get no preview card on X, and a blocked image URL means no thumbnail. We could not confirm this against X's current text.
Related bots
Other Social preview fetchers
- facebookexternalhit MetaMeta's fetcher that loads a page when its link is shared on Facebook, Instagram or Messenger, to build the preview.
- Slackbot SlackSlack's fetcher that expands links posted in channels into previews.
- TelegramBot TelegramReported to be the fetcher that appears in server logs when a link is shared in Telegram.
- LinkedInBot LinkedInThe fetcher reported to build link previews when someone shares a page on LinkedIn.
All Social Preview Fetchers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot