Install as a Cloudflare Worker

This works for any site, wherever it is hosted, as long as its domain is proxied by Cloudflare (the orange cloud on its DNS records). Add to Cloudflare does the same steps for you without a terminal.

Steps

1. Create an API token on the Account page and copy it.
2. In a new folder, save this wrangler.jsonc, with your domain in place of example.com. It adds a route for your domain and its www.:

{
  "name": "intercrawl",
  "main": "src/index.ts",
  "compatibility_date": "2026-09-30",
  "routes": ["example.com/*", "www.example.com/*"]
}

1. Save the install file in the same folder:

1. Store the token as a secret and deploy:

npx wrangler secret put INTERCRAWL_TOKEN
npx wrangler deploy

1. Press Check install on the Sites page.

Visitors' requests go straight to your site. Cloudflare gives the Worker each visitor's real IP address, so every crawler request is checked against its operator's published addresses.

Removing it

Delete the Worker in Cloudflare (Workers and Pages, the Worker, Settings, Delete). Its routes go with it.