# Crawler policy

Eight search and answer crawlers are allowed by name rather than by inheriting the wildcard. Five model-training crawlers are named with a commented-out `Disallow`, which leaves them under the wildcard — neither allowed explicitly nor blocked, because that decision belongs to the operator and has not been settled. `/api/` is disallowed for everyone.

- **Status:** Available
- **Audience:** developer
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/developers/crawler-policy/

## The decision table

Three groups, three different reasons. The file is served by the application rather than by a reverse proxy, so it is the same in every environment instead of correct only where somebody remembered to configure it.

| User agent | Kind | Decision | Why |
|---|---|---|---|
| `Googlebot` | Search | Allow, explicitly | Google Search, including AI Overviews and AI Mode |
| `Bingbot` | Search | Allow, explicitly | Bing Search and Copilot grounding |
| `OAI-SearchBot` | Search | Allow, explicitly | Surfaces sites in ChatGPT search. Not training |
| `Claude-SearchBot` | Search | Allow, explicitly | Improves Claude search result quality |
| `PerplexityBot` | Search | Allow, explicitly | Surfaces sites in Perplexity answers |
| `ChatGPT-User` | On a person's behalf | Allow, explicitly | A person in ChatGPT asked for this page |
| `Claude-User` | On a person's behalf | Allow, explicitly | A person in Claude asked for this page |
| `Perplexity-User` | On a person's behalf | Allow, explicitly | A person in Perplexity asked for this page |
| `GPTBot` | Training | Owner decision — left open | OpenAI foundation-model training |
| `ClaudeBot` | Training | Owner decision — left open | Anthropic model development |
| `Google-Extended` | Training | Owner decision — left open | Gemini / Vertex model training |
| `Applebot-Extended` | Training | Owner decision — left open | Apple foundation-model training |
| `CCBot` | Training | Owner decision — left open | Common Crawl, a training-data source |

The wildcard block above them all reads `Allow: /` with `Disallow: /api/`. `/app` is deliberately *not* disallowed: it carries a `noindex` meta tag, and a crawler has to be able to fetch a page to read the tag that tells it not to index the page.

## Why the allowed ones are written out one by one

Every one of those eight would already be allowed by the wildcard. Naming them changes nothing mechanically and quite a lot editorially: the decision becomes visible, and a later change to the wildcard cannot silently withdraw permission from a crawler somebody deliberately let in.

It also separates two things that are usually conflated. `OAI-SearchBot` and `ChatGPT-User` are different agents with different purposes — one builds an index, the other fetches a page because a person asked a question that needs it. Blocking the second does not protect anything; it breaks a person who is trying to read the documentation through a tool they already use.

The file ends with `Sitemap: https://connectbyjbrh.com/sitemap.xml`, which is the other half of the invitation: the policy says who may crawl, the sitemap says what there is.

## The training decision, and the trade-off in full

The five training crawlers appear with their `Disallow: /` commented out and a note saying the decision is unsettled. That is an unusual thing to publish, and it is more honest than either alternative: a silent omission would read as an oversight, and a `Disallow` would claim a decision nobody has made.

| Choice | Gain | Cost |
|---|---|---|
| Leave open (today) | A model that has read the documentation describes the product using the published status words rather than guessing | The text can end up in model weights, with no attribution, no link back, and no practical way to withdraw it later |
| Uncomment the `Disallow` | Training corpora exclude the site going forward | Nothing already collected is recalled; and a model that has not read the corpus answers about the product from inference instead |

Two facts make the choice narrower than it first looks. Training and search are separate user agents at every vendor listed, so blocking training does not remove the site from that vendor's search product — which is the whole reason the two can be decided apart. And the corpus is deliberately written so that being read is mostly a gain: a page that says a capability is `foundation` is more useful in a model's answer than the confident invention it replaces.

To change it, uncomment the `Disallow: /` under the agents concerned. The file is generated into `webapp/robots.txt` and served from there.

## What this file is not

The header comment states the boundary: this file expresses crawl preference, and it is not access control. Everything private sits behind a session whatever robots.txt says, and `Disallow: /api/` is a request not to crawl the API, not a lock on it.

Compliance is voluntary. A crawler that ignores the file is not doing anything a network can prevent, which is why the interesting question is usually not what the file says but whether the agent claiming to be `GPTBot` is `GPTBot` at all. [Verifying a bot is who it claims](/developers/bot-verification/) is the companion page.

> **Note** The format is RFC 9309, the Robots Exclusion Protocol. Longest-match wins within a group, and a named group replaces the wildcard for that agent entirely rather than adding to it — which is why the allowed agents each repeat `Allow: /`.

## Questions

### Is my content used to train a model if I use Connect?

That is a different question from this page, which is about crawling a public website. On the product side, the capability registry lists 'Connect trains a model on customer data' among the claims that must never be made.

### Why is `/app` crawlable at all?

Because the instruction not to index it lives in a `noindex` meta tag on the page. Disallowing the path in robots.txt would stop a crawler fetching the page, which would stop it reading the tag — and a URL can still appear in results when it is disallowed but linked.

### Does allowing an answer engine mean my documentation appears in its answers?

It means the crawler is not asked to stay away. What a vendor does with an allowed page is the vendor's decision, and no page in this corpus claims otherwise.

## Related

- [RFC 9309 — the Robots Exclusion Protocol](https://connectbyjbrh.com/docs/protocols/rfc-9309/)
- [Verifying a bot is who it claims](https://connectbyjbrh.com/developers/bot-verification/)
- [llms.txt on this site](https://connectbyjbrh.com/developers/llms-txt-here/)
- [The machine-readable documentation](https://connectbyjbrh.com/developers/machine-manifests/)
- [Measuring whether AI systems can actually read your site](https://connectbyjbrh.com/research/measuring-ai-visibility/)

## What this page is based on

- `webapp/robots.txt` — the served file and its comments
- `docs-source/facts.py` — CRAWLERS, verified against vendor documentation on 2026-09-10
- `docs-source/facts.py` — PROTOCOLS['robots'] (RFC 9309)