Get started
How serving works
What happens when a crawler asks for one of your pages, from the first request to the fallback.
Your install (a middleware file, a Cloudflare Worker, a Vercel routing rule or DNS mode) looks at each request to your site. A request from a browser goes on to your site as before. A request from a known crawler is sent to Intercrawl's render API, which answers with the page as a browser would show it.
Which requests count as crawlers
A request is a crawler request when its user agent names a crawler Intercrawl knows. The crawler list shows them all. Where the crawler's operator publishes the addresses it crawls from, Intercrawl also checks the request's IP address against them:
| Result | When | What the request gets |
|---|---|---|
| Verified | The IP address is in the operator's published ranges | A rendered page |
| Unverified | The operator publishes no ranges, or the install could not report the address | A rendered page |
| Spoofed | The user agent names a crawler, but the address is not one of its operator's | Your own page, as a visitor would get it |
Where the page comes from
- Cache. Rendered pages are kept for 7 days. A crawler asking for a page that was rendered recently gets the stored copy at once.
- Fresh render. A page with no stored copy is rendered on request, within the time limit below.
- Re-rendering. A stored page older than 7 days is still served while a new render is made in the background. Intercrawl also renders a page again when its scripts, stylesheets, title, description or canonical change (checked at most once an hour per page), when your sitemap gives it a newer
lastmod, or when you ask for it through the re-cache API.
Phones and computers get separate renders: a crawler that identifies as mobile (Googlebot smartphone, for example) gets the page rendered at a phone's width.
The time limit and fail-open
A crawler never waits more than 10 seconds. If a render is not ready in time, or anything fails on Intercrawl's side, your own page is served instead and the answer carries X-Intercrawl-Source: origin_fallback. Your site keeps working exactly as it does without Intercrawl.
What a rendered page contains
- The HTML of the page after its JavaScript ran, as a browser at the same width saw it.
- Scripts removed, except structured data (JSON-LD).
- The status code your page asks for: a page can set
<meta name="prerender-status-code" content="404">(and for redirects<meta name="prerender-header" content="Location: https://example.com/new">), the same convention Prerender uses. - A corrected canonical link: a relative one is made absolute, and one that points at a preview host (
localhost, an IP address,*.vercel.app,*.netlify.app,*.pages.dev,*.workers.dev) is moved to your own domain.
Every answer says where it came from in its response headers. See Response headers.
Limits
Each site has at most 1,000 fresh renders per UTC day. Past that, stored pages are still served and other requests get your own page.
What Intercrawl keeps
The serving system keeps no log of requests: no URL, IP address or user agent of your visitors or crawlers. For each site and UTC day it stores only counts (renders, cache hits, fallbacks, and verified, unverified and spoofed crawler requests).