Measuring whether AI systems can actually read your site

Three measurements, three kinds of evidence#

The three are not stages of one funnel, and treating them as one is the specific error. A page can be crawled constantly and never cited. It can be cited and never produce a referral, because a good answer is one the reader does not need to click through. And a referral can arrive from a surface whose fetch never appeared in a log at all.

Crawl: the only one your server can see#

A user-agent string is a claim, not an identity — anything can send any string. A crawl measurement is only worth keeping if the agent is verified the way its vendor publishes, by reverse lookup or by the vendor's own address ranges. An unverified count of a popular bot name is a count of everything that wanted to look like it.

The named agents also divide by purpose, and the division matters more than the total. Search-oriented crawlers exist to surface the site in answers. User-triggered agents fetch a page because a person in a chat asked for that specific page. Training crawlers collect material for model development, and whether to allow them is a separate decision from either of the others.

• Count fetches by verified agent and by purpose, not as one figure. Growth in a training crawler and growth in a search crawler mean different things.
• Count the status codes too. A crawler that received a redirect chain, a 403 or a soft error fetched nothing useful, and that is invisible in a hit count.
• Check what was served. A page that renders its content only after script execution is a different measurement from a page whose text is in the HTML.
• Publishing a change promptly is its own lever: an IndexNow submission tells participating engines a URL changed, up to 10,000 URLs in one request, authenticated by a key file of 8–128 hexadecimal characters.

Citation: sampling, honestly labelled#

There is no log of citations. The systems that would produce one are other people's, they share no identifier with your site, and several of them answer without linking at all. What is available is asking a fixed set of questions and reading what comes back — which is a sample of one conversation, on one day, in one region, on one model version.

A sample is genuinely useful for one thing: it shows how a subject is being described, and whether the description matches what the pages actually say. It is not useful as a rate, and converting it into a percentage is the point at which the measurement starts flattering whoever ran it.

Referral: undercounts by design#

Referral is the measurement most likely to be trusted, because it comes from analytics that already exist. It is also the one with the most structural undercount. An answer that fully satisfies the reader produces no click by definition. A referrer header may be stripped, generic, or absent entirely. And a click that does arrive lands on whichever page the answering system decided was the answer, which may not be the page anybody worked on.

Used carefully it still says something no other measurement does: that a real person came, and what they did next. Used as *the* number, it systematically understates the surfaces that answer well and overstates the ones that answer badly enough to require a click.

Why collapsing them flatters#

Crawl is the easiest of the three to grow and the only one you can see, so any single combined figure drifts towards being crawl with a more impressive name. Publish more pages, submit them promptly, and the number goes up while nothing has been learned about whether a single answer anywhere was better for the existence of the site.

1. Report the three separately, always, with the kind of evidence attached to each.
2. Label the sample as a sample. A citation check with twelve questions is twelve observations, not a rate.
3. Keep the crawl side correct and boring: a robots file that says what it means, sitemaps that match the published page set in both directions, prompt submission of changes, and no private path advertised anywhere.
4. Treat a supplemental map as supplemental. The llms.txt convention is a community proposal rather than an IETF or W3C standard, and Google has stated that no Search system reads it — publish it as a map, never as a substitute for HTML, robots or sitemaps.
5. Spend the remaining effort on the pages being quotable at all, which is a writing problem rather than a measurement one.

Questions#

Related#

• Crawler policy
• Verifying a bot is who it claims
• llms.txt on this site
• Writing documentation that both a person and a retrieval system can use
• End-to-end workflows