Move from Prerender.io

Intercrawl's render API answers in Prerender's format, so a site that uses one of Prerender's middlewares (prerender-node, prerender_rails, the nginx or Apache configurations, or a community one) can move by changing two settings: the service address and the token. For how the two services differ, see Intercrawl and Prerender.io.

Before you switch

1. Create an account and add your site on the Sites page.
2. Verify your site. Until it is verified, the render API answers your middleware with 403, and Prerender's middlewares pass that answer on to the crawler.
3. Create an API token on the Account page, under API tokens. It starts with ic_live_ and is shown once.

Keep your Prerender.io account until the last step below.

Change the service address and token

The new values:

prerender-node (Express and other Node.js servers)

Set two environment variables where your app runs, and remove any prerenderToken or prerenderServiceUrl written in your code (they win over the environment):

PRERENDER_SERVICE_URL=https://render.intercrawl.com/
PRERENDER_TOKEN=ic_live_...

Or in code, with the token read from the environment:

app.use(
  require("prerender-node")
    .set("prerenderServiceUrl", "https://render.intercrawl.com/")
    .set("prerenderToken", process.env.PRERENDER_TOKEN),
);

Leave forwardHeaders off: Intercrawl needs only the crawler's user agent, which the middleware always sends.

nginx

In Prerender's nginx configuration, change the token header and the address, and use https:

proxy_set_header X-Prerender-Token ic_live_...;
proxy_ssl_server_name on;

if ($http_x_prerender) {
    set $prerender 0;
}

if ($prerender = 1) {
    set $prerender "render.intercrawl.com";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass https://$prerender;
}

• proxy_ssl_server_name on is needed: the render API is served over https only, and needs its name in the connection.
• The X-Prerender check stops a loop: Intercrawl's renderer sends X-Prerender: 1 when it loads your page.
• Prerender's sample configuration recognises only search and social crawlers. Add the AI crawlers you want served to its user agent list, for example gptbot|oai-searchbot|chatgpt-user|claudebot|perplexitybot. The crawler list has every one Intercrawl renders for.
• If nginx receives connections from the internet directly (not behind a CDN), also send the crawler's address so Intercrawl can check it against the operator's published ranges: proxy_set_header X-Intercrawl-Client-IP $remote_addr;

Other middlewares

prerender_rails, the Apache configuration and the community middlewares have the same two settings, usually as PRERENDER_SERVICE_URL and PRERENDER_TOKEN or an option of the same name. Set them to the values in the table above.

Prerender.io's CDN integrations

If Prerender.io runs at your CDN (a Cloudflare Worker, or a rule at Akamai or CloudFront), remove it there and use the Intercrawl install for your site instead: Add to Cloudflare or the Cloudflare Worker for a domain on Cloudflare, or Choose an install for the others.

Check the switch

Deploy, then ask for a page as a crawler:

curl -sI -A "Mozilla/5.0 (compatible; Twitterbot/1.0)" https://example.com/

The answer should carry X-Intercrawl-Source: fresh the first time and cache after that. See Response headers. Then run an audit from the dashboard to see what each crawler receives on your pages.

Re-cache calls

If your deploys or your CMS call Prerender.io's re-cache API, change the address and the token. The request body stays the same:

POST https://render.intercrawl.com/recache
Content-Type: application/json

{ "prerenderToken": "ic_live_...", "urls": ["https://example.com/pricing"] }

Up to 1,000 URLs per request. Re-renders count towards the site's 1,000 fresh renders a day.

What stays the same, and what changes

• The same: <meta name="prerender-status-code"> and <meta name="prerender-header" content="Location: ..."> set the status and redirect exactly as with Prerender.io. Scripts are removed from the rendered page, except structured data (JSON-LD).
• The cache starts empty. Each page is rendered the first time a crawler asks for it, within 10 seconds; a crawler whose render is not ready in time gets your own page. To fill the cache before crawlers arrive, send your most important URLs to the re-cache API.
• Cache lifetime. Renders are kept 7 days and renewed when a page changes. See How serving works.
• Crawler checks. Prerender's middlewares do not send the crawler's IP address, so those requests are marked unverified and still rendered. Intercrawl's own install files send it; see Choose an install if you want each request checked.

Close the Prerender.io account

When the audits show your crawlers getting Intercrawl's renders, remove your Prerender token from your settings and close the Prerender.io account.