XML sitemap

What it means

An XML sitemap is a file listing the addresses of a website intended for search engines. It can also give each address a last-modified date and, on multilingual sites, link the language versions. From it the search engine learns which pages exist and which have changed.

A sitemap does not guarantee indexing, but it helps, most of all on large sites, new sites with few inbound links, pages deep in the structure and frequently changing content. On a small, well-linked site the benefit is smaller.

The file is usually called sitemap.xml, and its address is given in robots.txt and in the webmaster tools. Large sites split it into several files, for example by section or language, and tie them together with a sitemap index.

How it works in practice

Our website has sitemaps split by section and language, with a separate file for the glossary in each language. The last-modified date changes only when the content actually changes, not on every deployment.

Only pages meant to be in the index belong in the sitemap: canonical, accessible and not redirected. URLs with parameters, pages marked noindex or redirected addresses do not belong, because they send search engines conflicting signals.

In a site audit we compare the number of URLs in the sitemap with the number of indexed pages in the webmaster tools. A large gap points to a technical problem, duplicate content or pages that are too weak.

What to watch out for

The most common mistake is a sitemap that is not updated after the site changes and lists old, non-existent addresses. The search engine wastes time on errors, and new pages take longer to reach the results.

The second risk is an artificial last-modified date that changes on every deployment. Search engines learn to distrust the field and stop using it. The date should reflect a real change in content.