# llms.txt and a sitemap

A sitemap is an established discovery protocol that search crawlers read and act on. `llms.txt` is a community proposal — not an IETF or W3C standard — and Google has said no Search system reads it. Connect publishes both, with the sitemap as the mechanism and `llms.txt` as a supplemental map, never as a substitute for HTML, robots or sitemaps.

- **Status:** Reference
- **Audience:** developer
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/docs/compare/llms-txt-vs-sitemap/

## What each one is

**Sitemap** — XML listing the URLs a site wants crawled, with last-modified information, referenced from `robots.txt`. Read and acted on by search crawlers. It aids discovery; it is not a ranking instruction.
**robots.txt** — **RFC 9309** — the standard that says which agents may fetch what. This is the file with actual authority over crawling behaviour.
**llms.txt** — A community proposal: a Markdown file offering a curated map of a site for language models. No standards body behind it, and no confirmed consumer among the major search systems.
**IndexNow** — A push protocol — tell participating engines a URL changed rather than waiting to be crawled. Version 1.0: up to 10,000 URLs per bulk POST, and a key file of 8–128 hexadecimal characters.

Three of those four have a defined consumer. One does not, and that is the entire difference.

## Why publish llms.txt at all

Because it costs almost nothing, it is honest about what it is, and a curated map of a thousand-page corpus is genuinely useful to anything that reads it. What it must not do is carry weight the mechanism cannot bear.

> **Careful** The failure to avoid is treating it as a substitute. A site that publishes `llms.txt` and neglects its sitemap, its HTML or its `robots.txt` has swapped a protocol that works for a file whose readership is unconfirmed.

The same applies to the other machine artefacts here — [Markdown alternates](/developers/markdown-mirrors/), the [documentation manifests](/developers/machine-manifests/) and the [search API](/developers/docs-search-api/). Each is an addition to a properly built HTML page, never a replacement for one. If a fact is only in the manifest, a reader cannot see it and neither can most machines.

## Who is allowed to read what

Access is a `robots.txt` question, and Connect's crawler policy makes a distinction that is worth copying: **search and user-initiated fetching are not the same thing as training**.

| Kind | Examples | Why they differ |
|---|---|---|
| Search and answer surfacing | Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot | They exist to make your pages findable; blocking them removes you from answers |
| A person asked for this page | ChatGPT-User, Claude-User, Perplexity-User | Someone is reading your page right now, through a different window |
| Model training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot | A separate, deliberate decision for the site owner — not the same trade as being findable |

Conflating the three is how sites accidentally remove themselves from AI answers while trying to opt out of training. See [crawler policy](/developers/crawler-policy/) for the current stance and [verifying a bot](/developers/bot-verification/) for checking that an agent is who it says.

## A working setup

1. Publish real HTML pages first, each answering one question, with a canonical URL.
   - Result: Everything else is a pointer to those. Pointers to nothing help nobody.
2. Maintain the sitemap as a build artefact and reference it from `robots.txt`.
   - Result: Discovery works through the mechanism that has a defined consumer.
3. Set `robots.txt` deliberately per agent kind, not with one rule for everything with 'bot' in the name.
   - Result: You keep search and user-initiated reading while making the training decision separately.
4. Add `llms.txt` last, as a map.
   - Result: It helps whatever reads it and costs nothing if nothing does.

> **Note** The gate that builds this documentation refuses a public page missing from the sitemaps, and a sitemap entry that is not a public page. Keeping the two in agreement is a build responsibility rather than a periodic clean-up.

## Questions

### Will llms.txt get my site into AI answers?

There is no evidence that it will, and Google has said no Search system reads it. What demonstrably helps is being crawlable, being readable, and answering the question on the page rather than around it.

### Do I still need a sitemap if my site is small?

A small, well-linked site is discoverable without one. The sitemap earns its place as a site grows, when pages are added faster than they are linked, and as the place `lastmod` is published — which is also what makes a push protocol like IndexNow worth wiring up.

### Does Connect publish these for its own documentation?

Yes — sitemaps, `robots.txt`, `llms.txt`, Markdown alternates and machine manifests, all generated as build artefacts from the same page records the HTML comes from. That is deliberate: two hand-maintained copies of the same map disagree within a month.

## Related

- [llms.txt](https://connectbyjbrh.com/docs/protocols/llms-txt/)
- [XML sitemaps](https://connectbyjbrh.com/docs/technology/sitemap/)
- [Comparisons and concepts](https://connectbyjbrh.com/docs/compare/)
- [Crawler policy](https://connectbyjbrh.com/developers/crawler-policy/)
- [robots.txt](https://connectbyjbrh.com/docs/technology/robots-txt/)
- [llms.txt on this site](https://connectbyjbrh.com/developers/llms-txt-here/)

## What this page is based on

- `docs-source/facts.py` PROTOCOLS — llms.txt, RFC 9309, IndexNow 1.0
- `docs-source/facts.py` CRAWLERS — verified against each vendor's documentation
- https://llmstxt.org/
- `tools/docs_check.py` — sitemap and public-page agreement