Using Intercrawl
Reading results
What each status means, for a page and a crawler, in scans and audits.
For every page, Intercrawl compares each crawler's view with the human view and gives one status per crawler. A status describes what was observed on the date shown; it says nothing about how a search engine or AI service ranks, indexes or cites the page.
Statuses
| Status | Meaning |
|---|---|
| Match | The crawler received the page a visitor sees. |
| Divergent | The crawler received less: under 90% of the visitor's words, more than 20% of the visitor's internal links missing, or a different title or canonical. |
| Blank | The crawler's page had almost no text (under 20 words, or under 5% of the visitor's) while the visitor's had at least 20. |
| Error | The status codes differed, or the crawler's request failed or timed out. |
| Blocked | Your robots.txt disallows the page for that crawler, so it was not fetched. |
| Refused | The crawler was turned away before the page was served while the visitor got the page: for GPTBot and ClaudeBot an HTTP 401, 403 or 429, and for any crawler a block marked by a CDN's own header. Where the response names the provider (Vercel or Cloudflare), the result says which. |
| Not compared | Intercrawl could not load the human view (its own browser failed, or the page was larger than the size limit), so there was nothing to compare against. |
Seeing what a crawler received
Open a page from the dashboard or from a scan's results (signed in) to see it side by side: the crawler's view on the left, the visitor's on the right.
- Screenshot. What a visitor's browser showed. For the crawler, Show what GPTBot saw (or Googlebot, or ClaudeBot) draws the HTML that crawler received, with JavaScript off, so a page that needs JavaScript shows as the empty page the crawler got. Each account can draw 20 a day (UTC); a picture already drawn is free to see again.
- Text. The visible text each one got, one line per paragraph or block. Lines only the visitor got are highlighted on the right; lines only the crawler got on the left.
- HTML. The HTML each one got, one tag per line, side by side with the differences highlighted.
Each side shows its first 5,000 lines. Pictures, text and HTML are kept with the scan or audit for 30 days.
Severity
Each difference also has a severity. High: a different status code, a blank page, a page that looks not-found only to the crawler, or under half the visitor's words. Medium: other missing text, missing links, a different title or canonical. Low: a missing canonical, or different meta robots tags (these never change the status).
A site is marked materially divergent when any compared page is Blank, Error, or Divergent with a medium or high difference.
Things that are not counted
- Cookie and consent banners from known consent managers count no words or links in any view.
- Pages blocked by
robots.txt, refused, or not compared are left out of the match counts: there is nothing comparable in them.