Verifying a bot is who it claims
Anything can send User-Agent: Googlebot. Identity comes from the address the request arrives from, checked against a list the vendor publishes on its own domain, or from forward-confirmed reverse DNS where the vendor supports it. Connect gates nothing on a claimed agent: the public surface is read-only and unauthenticated, so there is nothing to gate.
Why the header proves nothing#
A User-Agent header is a free-text field the client fills in. There is no signature over it, no registry that binds a string to an operator, and no cost to copying one. Every crawler-impersonation problem on the open web starts with somebody treating that string as an identity, and the impersonation is usually of an agent that is *allowed*, precisely because allowing by name is common.
The same applies in reverse and is easier to miss: an agent that genuinely is what it says can be blocked by a rule matching on a substring, because vendors run several agents whose names share a prefix. Google-Extended and Googlebot are different decisions, and a rule written against "Google" makes one of them by accident.
The two mechanisms#
- Published address lists
- The vendor publishes a JSON file of the address ranges its crawlers use. You fetch it on a schedule, keep the parsed ranges, and check the connecting address against them. Ranges change, so a list cached once and never refreshed becomes a source of false rejections.
- Forward-confirmed reverse DNS
- Take the connecting address, resolve it to a hostname (PTR), check the hostname ends in a domain the vendor documents, then resolve that hostname forward and confirm it returns the original address. The forward step is not optional: a PTR record alone is controlled by whoever holds the address block.
Address lists are cheaper per request and need refreshing. Reverse DNS needs no list and costs two lookups, which is why it is usually cached per address for a bounded period. Where a vendor supports both, either is sufficient; where it publishes only a list, that list is the only authority.
Where each vendor publishes#
One rule governs all of these: fetch the source from the vendor's own domain over HTTPS. A third-party aggregation of crawler ranges is a copy of an authority, and copies go stale in the direction that hurts — they keep withdrawn ranges and miss new ones.
| Agents | Published by | Form |
|---|---|---|
Googlebot, Google-Extended, and Google's user-triggered fetchers | Google, in its Search developer documentation | JSON range files, one per crawler family, plus documented reverse-DNS domains (googlebot.com, google.com) |
Bingbot | Microsoft, through its Bing webmaster documentation | A JSON range file, plus reverse DNS into search.msn.com |
GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI, on its own site | Separate JSON range files per agent — they are not interchangeable |
ClaudeBot, Claude-SearchBot, Claude-User | Anthropic, in its crawler documentation | A published address list |
PerplexityBot, Perplexity-User | Perplexity, in its documentation | A published address list |
Applebot, Applebot-Extended | Apple, in its support documentation | Documented ranges and reverse-DNS guidance |
CCBot | Common Crawl, on its own site | Crawler documentation; treat as unverified where no address authority is published |
Does Connect use this?#
Not as an access decision. The public documentation surface — /api/public/*, the MCP endpoint's documentation tools, the manifests, the Markdown alternates — is unauthenticated and read-only by design, and every byte it returns is already served to any anonymous visitor. There is no benefit to be gained by impersonating a crawler, because there is nothing an impersonator would get that an honest client does not.
What is enforced instead is per-address rate limiting: 240 requests a minute on /api/public/, 120 a minute on /mcp, and on the MCP endpoint a 403 for a present-but-invalid Origin. Those are load and browser-safety controls, and they apply to every caller regardless of what it calls itself.
If you are running your own site and reading this for the technique, the sensible order is: express preference in robots.txt, verify only where you intend to act on the result, and verify at the edge. Verification you do not act on is a log line that costs two DNS lookups.
Questions#
Is there a header a crawler can send that proves who it is?
None of the crawlers named in this site's policy signs its requests. Identity is established from the connecting address, which is why every vendor publishes addresses rather than a token.
How often should a published range list be refreshed?
Often enough that a newly added range does not cause days of false rejections — daily is a common choice — and with the previous list kept as a fallback if the fetch fails, so a vendor outage does not turn into a blocking incident on your side.
Can I just block everything that fails verification?
Only if you have verified every agent you want to admit, including ordinary browsers and monitoring, which fail every crawler check by definition. Verification answers 'is this who it claims', not 'is this welcome'.