robots.txt
/robots.txt is a plain-text file at the root of a host that tells automated clients which paths they may fetch. Since 2022 it has been a standard, RFC 9309, rather than a convention. It controls fetching. It does not control indexing, it does not hide anything, and it is obeyed only by clients that choose to obey it.
The misunderstanding that causes real damage#
Disallow means *do not fetch this*. It does not mean *do not list this*. A URL that is disallowed can still appear in a search index if other pages link to it, because the crawler learned the URL exists without ever reading the page. The listing is then built from the link text alone, which is the worst of both outcomes: the page is listed and nothing about it is accurate.
The same logic applies to anything you would rather nobody found. A robots file is a public document listing paths you have drawn attention to. It is a traffic-management tool, not an access-control one; authentication is what stops a fetch.
How a group is matched#
The file is a sequence of groups. Each group has one or more User-agent lines and then the rules that apply to them.
- A crawler obeys exactly one group: the one whose user-agent token is the most specific match for its own name. It does not merge that group with the
*group. - Matching is case-insensitive on the token and is a substring-style match on the product name, not a full user-agent string comparison.
- Within the chosen group, the most specific rule wins — the longest matching path — and
AllowbeatsDisallowwhen the two are the same length. Disallow:with an empty value allows everything.Disallow: /refuses everything. The difference is one character.Sitemap:is not part of any group. It is a file-level line, takes an absolute URL, and may appear several times.
Does Connect use robots.txt, and how is it served?#
Used, and served by the application itself rather than by a proxy in front of it. That is a deliberate choice with one practical consequence: the file is correct in every environment, because it is generated by the same deployment that serves the pages, and a preview or a staging host cannot inherit production's rules or lose its own.
Two decisions in that file are worth stating, because both look like mistakes to anyone auditing it quickly:
/appis not disallowed- The application is kept out of search results with a
noindexmeta tag on the page itself. Disallowing it would prevent the crawler ever fetching the page, so thenoindexwould never be seen, and the URL could be listed anyway from inbound links. Allowing the fetch is what makes the exclusion work. - The sitemap is named in the file
- A
Sitemap:line gives every crawler the entry point without a submission step, and it is the only part of the file that adds URLs rather than removing them.
Crawler policy is stated by name rather than left to a wildcard. Search and user-initiated agents — Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User — are allowed, because a person asking a question deserves the real page rather than a guess about it. Foundation-model training crawlers such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot are a separate category and a separate decision; the crawler policy page states the current position for each.
Serving it correctly#
| Requirement | Consequence of getting it wrong |
|---|---|
Exactly at /robots.txt on each host and scheme | A file on www. does not apply to the bare domain. They are different hosts. |
text/plain, UTF-8 | Served as HTML or with a download disposition, it may not be parsed at all. |
| HTTP 200 | A 5xx is commonly treated as *disallow everything*, temporarily. A sustained 5xx can stop crawling of the whole site. |
| Under 500 KiB | RFC 9309 permits parsers to stop reading there; rules past the cut-off silently vanish. |
| No authentication | A robots file behind a login is a robots file that does not exist. |
A 404 is a valid and safe answer: it means no restrictions. The dangerous responses are the ones that are neither a file nor an absence — a redirect chain, a soft 404 rendering an HTML error page with status 200, or an intermittent 503 during a deploy.
Questions#
Will disallowing a page remove it from search results?
No, and it can make removal impossible. The crawler stops fetching the page, which means it never sees a noindex instruction, while the URL can still be listed from links elsewhere. To remove a page, allow the fetch and serve noindex; once it has dropped out, disallowing is optional.
Does robots.txt keep private data safe?
It does the opposite. The file is public, and listing a path there tells everyone that the path exists and that you would rather they did not look. Anything that must not be read needs authentication or must not be served. Compliance with robots.txt is voluntary and unenforced.
Should a site block AI crawlers in robots.txt?
That depends on which crawler, because they do different jobs. A user-initiated fetch — someone in an assistant asked for your page — and a search crawler that surfaces your site in answers are both traffic. A foundation-model training crawler is a separate decision with different trade-offs, and the two categories should never be collapsed into one rule.