Crawler policy
Eight search and answer crawlers are allowed by name rather than by inheriting the wildcard. Five model-training crawlers are named with a commented-out Disallow, which leaves them under the wildcard — neither allowed explicitly nor blocked, because that decision belongs to the operator and has not been settled. /api/ is disallowed for everyone.
The decision table#
Three groups, three different reasons. The file is served by the application rather than by a reverse proxy, so it is the same in every environment instead of correct only where somebody remembered to configure it.
| User agent | Kind | Decision | Why |
|---|---|---|---|
Googlebot | Search | Allow, explicitly | Google Search, including AI Overviews and AI Mode |
Bingbot | Search | Allow, explicitly | Bing Search and Copilot grounding |
OAI-SearchBot | Search | Allow, explicitly | Surfaces sites in ChatGPT search. Not training |
Claude-SearchBot | Search | Allow, explicitly | Improves Claude search result quality |
PerplexityBot | Search | Allow, explicitly | Surfaces sites in Perplexity answers |
ChatGPT-User | On a person's behalf | Allow, explicitly | A person in ChatGPT asked for this page |
Claude-User | On a person's behalf | Allow, explicitly | A person in Claude asked for this page |
Perplexity-User | On a person's behalf | Allow, explicitly | A person in Perplexity asked for this page |
GPTBot | Training | Owner decision — left open | OpenAI foundation-model training |
ClaudeBot | Training | Owner decision — left open | Anthropic model development |
Google-Extended | Training | Owner decision — left open | Gemini / Vertex model training |
Applebot-Extended | Training | Owner decision — left open | Apple foundation-model training |
CCBot | Training | Owner decision — left open | Common Crawl, a training-data source |
The wildcard block above them all reads Allow: / with Disallow: /api/. /app is deliberately *not* disallowed: it carries a noindex meta tag, and a crawler has to be able to fetch a page to read the tag that tells it not to index the page.
Why the allowed ones are written out one by one#
Every one of those eight would already be allowed by the wildcard. Naming them changes nothing mechanically and quite a lot editorially: the decision becomes visible, and a later change to the wildcard cannot silently withdraw permission from a crawler somebody deliberately let in.
It also separates two things that are usually conflated. OAI-SearchBot and ChatGPT-User are different agents with different purposes — one builds an index, the other fetches a page because a person asked a question that needs it. Blocking the second does not protect anything; it breaks a person who is trying to read the documentation through a tool they already use.
The file ends with Sitemap: https://connectbyjbrh.com/sitemap.xml, which is the other half of the invitation: the policy says who may crawl, the sitemap says what there is.
The training decision, and the trade-off in full#
The five training crawlers appear with their Disallow: / commented out and a note saying the decision is unsettled. That is an unusual thing to publish, and it is more honest than either alternative: a silent omission would read as an oversight, and a Disallow would claim a decision nobody has made.
| Choice | Gain | Cost |
|---|---|---|
| Leave open (today) | A model that has read the documentation describes the product using the published status words rather than guessing | The text can end up in model weights, with no attribution, no link back, and no practical way to withdraw it later |
Uncomment the Disallow | Training corpora exclude the site going forward | Nothing already collected is recalled; and a model that has not read the corpus answers about the product from inference instead |
Two facts make the choice narrower than it first looks. Training and search are separate user agents at every vendor listed, so blocking training does not remove the site from that vendor's search product — which is the whole reason the two can be decided apart. And the corpus is deliberately written so that being read is mostly a gain: a page that says a capability is foundation is more useful in a model's answer than the confident invention it replaces.
To change it, uncomment the Disallow: / under the agents concerned. The file is generated into webapp/robots.txt and served from there.
What this file is not#
The header comment states the boundary: this file expresses crawl preference, and it is not access control. Everything private sits behind a session whatever robots.txt says, and Disallow: /api/ is a request not to crawl the API, not a lock on it.
Compliance is voluntary. A crawler that ignores the file is not doing anything a network can prevent, which is why the interesting question is usually not what the file says but whether the agent claiming to be GPTBot is GPTBot at all. Verifying a bot is who it claims is the companion page.
Questions#
Is my content used to train a model if I use Connect?
That is a different question from this page, which is about crawling a public website. On the product side, the capability registry lists 'Connect trains a model on customer data' among the claims that must never be made.
Why is /app crawlable at all?
Because the instruction not to index it lives in a noindex meta tag on the page. Disallowing the path in robots.txt would stop a crawler fetching the page, which would stop it reading the tag — and a URL can still appear in results when it is disallowed but linked.
Does allowing an answer engine mean my documentation appears in its answers?
It means the crawler is not asked to stay away. What a vendor does with an allowed page is the vendor's decision, and no page in this corpus claims otherwise.