# XML sitemaps An XML sitemap is a list of URLs you consider worth crawling, with optional metadata about each. It helps a crawler find pages that links alone would reach slowly or not at all. It is a suggestion, not an instruction: nothing in a sitemap is guaranteed to be crawled, and nothing absent from one is excluded. - **Status:** Reference - **Audience:** both, developer - **Last verified:** 2026-09-10 - **Canonical:** https://connectbyjbrh.com/docs/technology/sitemap/ ## What a sitemap is actually for The value is concentrated in a few situations, and outside them a sitemap changes very little. Knowing which situation you are in decides how much effort the file deserves. - **Deep or weakly linked pages.** A page four clicks from anywhere, or reachable only through a filter, may be found slowly. The sitemap shortens that. - **Large corpora.** A thousand pages published at once are discovered far faster from a list than from crawling the link graph outward. - **Change signalling.** A truthful `lastmod` tells a crawler which pages are worth re-fetching, which matters more as a site grows. - **New sites.** With few inbound links there is little for a crawler to follow, and the sitemap is close to the only entry point. What a sitemap does not do: it does not cause indexing, it does not override a `noindex`, it does not consolidate duplicates and it does not confer priority over anyone else's pages. The `` and `` elements in the original protocol are widely ignored by the major search engines, and writing them carefully is effort spent on nothing. ## lastmod, and the reason it is usually wrong `lastmod` is the one element still read closely, and it is trusted only while it is accurate. A file that stamps every URL with today's date each time it is generated is claiming the entire site changed today. After a few cycles of that the element is discounted for the whole domain, and you have destroyed the one signal you had. | Change | Update lastmod? | |---|---| | The page's substantive content changed | Yes | | A typo was corrected in a heading | Yes, if you are honest about it; the date is a fact, not a claim of significance | | The site-wide navigation or footer changed | No. Every page would move, and none of them changed | | The build ran again with no content difference | No. This is the failure mode above | | A page was rendered with a new template | No, unless the content it presents is different | The date must be a valid W3C datetime — a full date, or a date and time with a timezone offset. A date in the future is treated as invalid, and a timezone-naïve timestamp on a site that publishes across regions will be interpreted in a way you did not choose. ## Sizes, index files and splitting A single sitemap file holds at most **50,000 URLs** and at most **50 MiB uncompressed**. Past either limit the file is invalid, not truncated. A sitemap index file is a list of sitemaps and is subject to the same 50,000 and 50 MiB limits on its own entries; indexes may not nest. ```xml https://connectbyjbrh.com/sitemap-docs.xml 2026-09-10 ``` Split by section rather than by arithmetic wherever you can. A sitemap per area of the site means the `lastmod` on the index entry is itself informative, and it makes a coverage problem attributable — if one section is under-crawled you can see which. > **Note** Every URL in a sitemap must be on the same host as the sitemap unless you have proved ownership of the other host. A cross-host entry is usually dropped in silence rather than reported. ## Does Connect use a sitemap, and how is it built? **Used, and generated by the documentation build from the same page records the pages themselves are rendered from.** Like `robots.txt`, it is served by the application rather than by a proxy, so a deployment carries its own sitemap and no environment can serve another environment's list. Because the list and the pages come from one source, the build can hold a rule that a hand-maintained file never sustains: **a public page missing from the sitemaps is a build failure, and a sitemap entry that is not a public page is also a build failure.** The check runs on every build rather than on an audit schedule, which is why the two can never drift apart by more than one commit. The application at `/app` is not listed, because it is not a public documentation page and it carries a `noindex` meta tag. Listing a `noindex` URL in a sitemap is a contradiction — an invitation to crawl something you have asked not to be indexed — and the build refuses it for that reason rather than as a style rule. ## Questions ### Does a sitemap guarantee a page will be indexed? No. It makes discovery cheap and fast; whether a page is indexed depends on whether it is judged worth indexing. A page that is in the sitemap and not in the index has usually been seen and passed over, which is a content question rather than a crawling one. ### Should URLs that redirect be listed? No. A sitemap should contain final, canonical, 200-returning URLs only. Listing a redirect wastes a fetch and sends a mixed signal about which address you prefer — the same conflict that makes a canonical element get ignored. ### How does a crawler find the sitemap? Two ways that cost nothing to do together: a `Sitemap:` line in `robots.txt`, which every compliant crawler reads, and a submission in whichever webmaster tool you use. The robots line is the one that works for crawlers you have never heard of. ## Related - [robots.txt](https://connectbyjbrh.com/docs/technology/robots-txt/) - [IndexNow](https://connectbyjbrh.com/docs/technology/indexnow/) - [Canonical URLs](https://connectbyjbrh.com/docs/technology/canonical-url/) - [The machine-readable documentation](https://connectbyjbrh.com/developers/machine-manifests/) - [Treating documentation as a build artefact](https://connectbyjbrh.com/research/documentation-as-a-build/) ## What this page is based on - Sitemaps XML protocol 0.9, https://www.sitemaps.org/protocol.html - Google Search Central — sitemaps and lastmod, https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap - Connect documentation gate — `tools/docs_check.py`, the public-page/sitemap consistency refusal - Connect capability registry (docs-source/facts.py) — public_docs, docs_manifests