# RFC 9309 — the Robots Exclusion Protocol RFC 9309 standardised the Robots Exclusion Protocol that had been a de facto convention since 1994. It defines `user-agent`, `allow` and `disallow` groups, how a crawler picks the matching rule, and what to do when the file cannot be fetched. It is a crawl preference, never access control. - **Status:** Reference - **Audience:** developer, both - **Last verified:** 2026-09-10 - **Canonical:** https://connectbyjbrh.com/docs/protocols/rfc-9309/ ## What the standard actually says **Groups** — Records are grouped by `user-agent`. A crawler picks the group matching its own product token, case-insensitively, or the `*` group if there is no specific one — not both. **Rule matching** — The most specific rule wins, measured by the length of the matched path. If an `allow` and a `disallow` match with equal specificity, `allow` wins. **Wildcards** — `*` matches any sequence and `$` anchors the end of the path. These were widespread practice before the RFC and are now in it. **Case** — Paths are case-sensitive. `/Docs/` and `/docs/` are different rules, whatever the file system underneath does. **Scope** — One file per origin — scheme, host and port. `https://example.net` and `http://example.net` do not share a policy. **Size** — A crawler is required to parse at least the first 500 KiB. Beyond that it may stop, so nothing important belongs at the bottom of a very long file. Status-code handling is the part that most repays reading, because it is the opposite of what people assume. A `4xx` — the file is absent — means there are no restrictions and the crawler may fetch everything. A `5xx` means the policy is *unknown*, and a crawler should assume a complete disallow until it can read the file. A broken server is therefore treated as more restrictive than a missing file. ## The parts commonly misquoted - **`Crawl-delay` is not in the standard.** Some crawlers honour it, several ignore it, and none is obliged to. Rate control that must hold belongs in the server. - **`Sitemap:` is not defined by RFC 9309 either.** It is a widely supported extension, understood by the major search crawlers, and worth including — but it is convention, not conformance. - **`Disallow` does not remove a page from search results.** It stops a compliant crawler fetching the page; a URL discovered elsewhere can still appear, described only by its links. Keeping a page out of an index needs a `noindex` the crawler is *allowed to fetch and read*. - **It is not access control.** The file is a request to well-behaved software. Anything that must not be reached needs authentication. - **An empty `Disallow:` means allow.** It is a common typo to write `Disallow:` intending to block everything; `Disallow: /` is the rule that does that. > **Careful** The `noindex` trap catches careful people. Disallowing a path in `robots.txt` prevents a crawler from ever seeing the `noindex` tag on the page, so the two directives together achieve less than the tag alone. ## Does Connect use RFC 9309? Yes. `robots.txt` is served by the application rather than by whatever reverse proxy happens to be in front of it, so it is correct in every environment rather than only in production. The file states in its own comments that it expresses crawl preference and is not access control: everything private is behind a session whatever the file says. The policy has three parts. The wildcard group allows everything and disallows `/api/`. Search and answer crawlers — Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User — are named and allowed explicitly, so the decision is visible rather than inherited. Model-training crawlers — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot — are named with a commented-out `Disallow`, because training is a separate decision from search and the trade-off is written up rather than made silently. `/app` is deliberately **not** disallowed. The application carries a `noindex` meta tag, and a crawler has to be able to fetch the page to read it — which is exactly the trap described above. A `Sitemap:` line points at the site's sitemap. Connect does not itself crawl the web, so it is a publisher of this protocol and never a consumer of it. Its prospect research reads sources it is given rather than operating a crawler. ## Checking your own file 1. Fetch it over the exact origin you care about, including scheme and port. 2. Confirm it is `text/plain` and returns `200`. A soft error page with status `200` is read as a policy and parsed as nonsense. 3. Check what a `5xx` does to you: if the file becomes unreachable during an incident, compliant crawlers stop entirely. 4. Test the rule you rely on against the longest-match rule, not against your intent — specificity is measured in characters, and an `Allow` deeper than your `Disallow` re-opens the path. 5. Confirm that anything carrying `noindex` is fetchable. ## Questions ### Does robots.txt keep a page private? No. It asks well-behaved crawlers not to fetch a path. It is a public file that advertises the paths you care about, and it binds nobody. Privacy comes from authentication — see [Security and isolation](/docs/security/). ### Why are training crawlers listed but not blocked? Because that decision is separate from search and has not been settled. Naming them with a commented-out rule makes the choice visible and easy to change in one place; the reasoning is at [Crawler policy](/developers/crawler-policy/). ### Should I disallow my application's URLs? Usually not, if those pages carry `noindex`. Disallowing them stops the crawler reading the tag, so the page can still be listed without ever having been fetched. Allow the fetch and let the tag do the work. ## Related - [llms.txt](https://connectbyjbrh.com/docs/protocols/llms-txt/) - [Crawler policy](https://connectbyjbrh.com/developers/crawler-policy/) - [Verifying a bot is who it claims](https://connectbyjbrh.com/developers/bot-verification/) - [Well-known URIs](https://connectbyjbrh.com/docs/protocols/well-known-uris/) - [Measuring whether AI systems can actually read your site](https://connectbyjbrh.com/research/measuring-ai-visibility/) ## What this page is based on - https://www.rfc-editor.org/rfc/rfc9309 - `webapp/robots.txt` — the served policy and its comments - `docs-source/facts.py` — CRAWLERS and PROTOCOLS['robots']