Crawler policy

The Crawler policy page is a matrix: AI crawlers down the side, path rules across the top, and in each cell Allow, Block or Charge.

Building a policy

• The first rule, /*, covers your whole site. Add path rules such as /blog/*; for each request the first matching rule decides, and a cell you leave empty follows /*.
• Block and Allow work on any rule. Charge is set on the site-wide row only, with one price per request for the zone.
• Changes are saved as you make them, as a draft, and marked pending until applied.

Applying it

Apply needs your domain on Cloudflare. You approve Intercrawl on Cloudflare's own page at each Apply; Intercrawl writes the policy and gives the access back, keeping none.

• Blocks become one WAF custom rule on your zone, described "Intercrawl crawler policy". Your other rules are never touched.
• Charges use Cloudflare's Pay Per Crawl, where your Cloudflare account has it (it is in Cloudflare's beta). Payments are collected by Cloudflare, never by Intercrawl. Where it is not available, Charge shows "Pay Per Crawl beta pending".

Until paid plans open, every account can build a policy as a draft; applying it comes with the Growth plan.