llms.txt and a sitemap
A sitemap is an established discovery protocol that search crawlers read and act on. llms.txt is a community proposal — not an IETF or W3C standard — and Google has said no Search system reads it. Connect publishes both, with the sitemap as the mechanism and llms.txt as a supplemental map, never as a substitute for HTML, robots or sitemaps.
What each one is#
- Sitemap
- XML listing the URLs a site wants crawled, with last-modified information, referenced from
robots.txt. Read and acted on by search crawlers. It aids discovery; it is not a ranking instruction. - robots.txt
- RFC 9309 — the standard that says which agents may fetch what. This is the file with actual authority over crawling behaviour.
- llms.txt
- A community proposal: a Markdown file offering a curated map of a site for language models. No standards body behind it, and no confirmed consumer among the major search systems.
- IndexNow
- A push protocol — tell participating engines a URL changed rather than waiting to be crawled. Version 1.0: up to 10,000 URLs per bulk POST, and a key file of 8–128 hexadecimal characters.
Three of those four have a defined consumer. One does not, and that is the entire difference.
Why publish llms.txt at all#
Because it costs almost nothing, it is honest about what it is, and a curated map of a thousand-page corpus is genuinely useful to anything that reads it. What it must not do is carry weight the mechanism cannot bear.
The same applies to the other machine artefacts here — Markdown alternates, the documentation manifests and the search API. Each is an addition to a properly built HTML page, never a replacement for one. If a fact is only in the manifest, a reader cannot see it and neither can most machines.
Who is allowed to read what#
Access is a robots.txt question, and Connect's crawler policy makes a distinction that is worth copying: search and user-initiated fetching are not the same thing as training.
| Kind | Examples | Why they differ |
|---|---|---|
| Search and answer surfacing | Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot | They exist to make your pages findable; blocking them removes you from answers |
| A person asked for this page | ChatGPT-User, Claude-User, Perplexity-User | Someone is reading your page right now, through a different window |
| Model training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot | A separate, deliberate decision for the site owner — not the same trade as being findable |
Conflating the three is how sites accidentally remove themselves from AI answers while trying to opt out of training. See crawler policy for the current stance and verifying a bot for checking that an agent is who it says.
A working setup#
Publish real HTML pages first, each answering one question, with a canonical URL.
Result Everything else is a pointer to those. Pointers to nothing help nobody.
Maintain the sitemap as a build artefact and reference it from
robots.txt.Result Discovery works through the mechanism that has a defined consumer.
Set
robots.txtdeliberately per agent kind, not with one rule for everything with 'bot' in the name.Result You keep search and user-initiated reading while making the training decision separately.
Add
llms.txtlast, as a map.Result It helps whatever reads it and costs nothing if nothing does.
Questions#
Will llms.txt get my site into AI answers?
There is no evidence that it will, and Google has said no Search system reads it. What demonstrably helps is being crawlable, being readable, and answering the question on the page rather than around it.
Do I still need a sitemap if my site is small?
A small, well-linked site is discoverable without one. The sitemap earns its place as a site grows, when pages are added faster than they are linked, and as the place lastmod is published — which is also what makes a push protocol like IndexNow worth wiring up.
Does Connect publish these for its own documentation?
Yes — sitemaps, robots.txt, llms.txt, Markdown alternates and machine manifests, all generated as build artefacts from the same page records the HTML comes from. That is deliberate: two hand-maintained copies of the same map disagree within a month.