Using Intercrawl
Crawlers Intercrawl recognises
The crawlers that get a rendered page, and how each one's requests are checked.
A request whose user agent names one of these crawlers is a crawler request and gets a rendered page. Where the operator publishes the IP addresses it crawls from, Intercrawl checks each request against them, and a request from any other address gets your own page instead. See How serving works.
| Crawler | Operator | How a request is checked |
|---|---|---|
| Googlebot | User agent and the operator's published IP ranges | |
| Google special crawler | User agent and the operator's published IP ranges | |
| Google user fetcher | User agent and the operator's published IP ranges | |
| Bingbot | Microsoft | User agent and the operator's published IP ranges |
| GPTBot | OpenAI | User agent and the operator's published IP ranges |
| OAI-SearchBot | OpenAI | User agent and the operator's published IP ranges |
| ChatGPT-User | OpenAI | User agent and the operator's published IP ranges |
| ClaudeBot | Anthropic | User agent and the operator's published IP ranges |
| Claude-SearchBot | Anthropic | User agent and the operator's published IP ranges |
| Claude-User | Anthropic | User agent and the operator's published IP ranges |
| PerplexityBot | Perplexity | User agent and the operator's published IP ranges |
| Perplexity-User | Perplexity | User agent and the operator's published IP ranges |
| Applebot | Apple | User agent and the operator's published IP ranges |
| DuckDuckBot | DuckDuckGo | User agent only (no published IP ranges) |
| YandexBot | Yandex | User agent only (no published IP ranges) |
| facebookexternalhit | Meta | User agent only (no published IP ranges) |
| Twitterbot | X | User agent only (no published IP ranges) |
| LinkedInBot | User agent only (no published IP ranges) | |
| Slackbot | Slack | User agent only (no published IP ranges) |
| Discordbot | Discord | User agent only (no published IP ranges) |
| Meta | User agent only (no published IP ranges) | |
| TelegramBot | Telegram | User agent only (no published IP ranges) |
The published address lists are refreshed every day. This list itself was last reviewed 12 October 2026 against the community list ai.robots.txt and each operator's own published pages; a monthly check compares the two and emails Intercrawl's operator with anything new to review.
Not answered here? Write to hello@intercrawl.com.