Help
Response headers
The X-Intercrawl headers on a crawler's answer, and what each value means.
Request a page of your site with a crawler's user agent and look at the response headers to see what Intercrawl did. For example:
curl -sI -A "Mozilla/5.0 (compatible; Twitterbot/1.0)" https://example.com/
X-Intercrawl-Source
| Value | Meaning |
|---|---|
cache | A stored render was served. |
fresh | The page was rendered for this request. |
origin | Not a crawler (a visitor, or a crawler whose address is not its operator's), so your own page was served. |
origin_fallback | Intercrawl had no render in time (the 10-second limit, a busy renderer, a daily limit, or an error on its side), so your own page was served. |
disconnected | A Vercel rule from an ended connection: your own page, until you remove the rule. |
error | The request was refused before anything was fetched (no token, an unknown token, or a URL on no verified site). |
With the install files (Next.js, Vercel, Netlify, Cloudflare Worker, Cloudflare Pages), only cache and fresh answers come from Intercrawl. For anything else the file steps aside and your site answers the request itself, with no Intercrawl header. A crawler request answered without any X-Intercrawl-Source header means your own page was served.
X-Intercrawl-Verified
| Value | Meaning |
|---|---|
verified | The request came from an address in the crawler operator's published ranges. |
unverified | The operator publishes no ranges, or the install could not report the address. |
spoofed | The user agent names a crawler, but the address is not its operator's. |
human | Not a crawler. |
audit | One of Intercrawl's own audit requests, signed. |
X-Intercrawl-Rendered-At
When the served render was made, in UTC.
Not answered here? Write to hello@intercrawl.com.Next: Troubleshooting