A search crawler fetching a page
A crawler gets what a reader gets: the whole page as HTML, served without JavaScript, a cookie or a sign-in. Policy is set by purpose rather than by company — search crawling and user-initiated fetches are allowed, model-training crawlers are a deliberate decision recorded separately — and identity is settled by verifying the fetch, never by trusting the user-agent string.
What comes back#
- The complete page as server-rendered HTML. There is no client-side rendering step to wait for, and no content that appears only after script runs.
- A Markdown alternate of the same page, generated from the same record, for clients that would rather have text.
- The manifests and discovery files: the corpus as JSON, the capability status document, the sitemap set, and the short discovery map.
- The same content for every requester. There is no personalisation, no geographic variation of the text, and nothing behind an interstitial.
What does not come back is anything about a business using Connect. The documentation describes the product; workspace data lives behind authentication and is not part of any crawlable surface. A crawler cannot reach a conversation, a contact or a call, because those are not published anywhere for it to reach.
One fetch, stage by stage#
- Trigger — a crawler schedules a fetch, or a person in an AI product asks a question that makes their client fetch a page on their behalf.
- External event — an HTTP GET arrives with a user-agent string and, for a verifiable bot, an address that can be checked against the vendor.
- Authentication and workspace resolution — none, and that is the design: a public page that needed a session would be a public page nobody could read.
- Request — the path resolves to a generated file. Robots rules apply by purpose, per RFC 9309, and are published rather than implied.
- Canonical record — none is written. A crawl creates no row and no contact.
- Classification — the agent is matched against the published policy: search, user-initiated, or model training.
- Knowledge and rules — the crawler policy and the sitemap set are the rules; both are public documents so a crawler operator can check them without asking.
- Autonomy and approval — not applicable. Nothing is sent, spent or decided on a person's behalf by serving a page.
- Action — the static file is served, with its canonical link and its structured data.
- Result — 200 with the full page. Anything else is a failure worth investigating, including a 200 that carries a shell.
- Relationship and timeline — none. A crawler is not a contact and is never resolved to one.
- Audit — server logs only. Reachability is proved by fetching as the agent, not by reading a log after the fact.
Policy by purpose, not by company#
| Purpose | Examples | Stance |
|---|---|---|
| Search indexing | Googlebot, Bingbot and the search crawlers of AI answer products | Allowed — the corpus exists to be found |
| User-initiated fetch | A client fetching one page because a person in an AI product asked for it | Allowed — this is a reader with a different browser |
| Model training | Foundation-model training crawlers | A separate decision, recorded as the operator's rather than assumed either way |
Splitting by purpose matters because one company often runs several agents with different jobs, and treating them as one thing produces a policy nobody can implement. Allowing a search crawler while deciding separately about a training crawler is a coherent position; allowing or blocking a whole vendor is usually not.
Proving a bot is what it says#
A user-agent string is a claim, not evidence — anyone can send any string. Verification means checking that the request genuinely came from the operator it names, by the method that operator publishes for the purpose. Treating the string as identity is how a policy that looks strict turns out to allow anything that copies the right text.
The practical consequence for anyone operating a crawler: identify yourself honestly, come from the addresses your own documentation names, and respect the published rules. The practical consequence for anyone reading logs: count verified fetches, not user-agent strings, before concluding anything about who is reading the site.
When a crawler cannot get the page#
- 401 or 403
- Almost always an edge rule rather than the application — a bot filter or a security product deciding before the request reaches anything that knows about documentation.
- 429
- Rate limiting, usually at the edge as well. The fix is a rule that recognises the agent, not a change to the page.
- 503 with a challenge body
- A JavaScript or CAPTCHA interstitial. A crawler cannot solve it, so the page is invisible however good it is.
- A redirect to a sign-in
- The page is not public after all, which is a routing fault worth fixing immediately.
- 200 with an empty shell
- The page needs script to render. Nothing in this corpus does, so this would mean something between the crawler and the file replaced it.
Every one of these is invisible from inside the application, which is why reachability is checked by fetching as each agent and comparing what came back.
Questions#
Does this site require JavaScript to read a page?
No. Pages are generated as static HTML and served as files. That is partly for crawlers and partly for readers on poor connections; a page whose text arrives only after script has run is a page that some readers and most machines never see.
Is a user-initiated fetch the same as crawling?
No, and the distinction matters for policy. A crawler decides for itself what to fetch and when; a user-initiated fetch happens because a person asked their AI product a question. The second is closer to a reader clicking a link than to indexing.
Can a crawler reach anything about a business using Connect?
No. Workspace data is behind authentication, is filtered by three independent layers, and is not published in any form a crawler could request. The public corpus describes the product, not any customer.