Noindex

What it means

Noindex is an instruction telling search engines not to add a specific page to their index or show it in search results. It is set either with a meta robots tag in the page code or with an X-Robots-Tag header in the server response, which also works for files such as PDFs or images.

For a search engine to see the noindex instruction, it must first visit the page. That is why noindex must not be combined with a crawl ban in robots.txt: if the crawler may not enter the page, it never reads the instruction, and the page can still appear in the index via links from other sites.

Noindex is often combined with nofollow, which tells the crawler not to follow links on the page. For ordinary use noindex alone is enough; the crawler can still follow the links and discover other content through them.

How it works in practice

Noindex is used for pages that should be available to people but have no place in results: internal search, thank-you pages after a form is sent, an online store’s basket and checkout, filtered pages, login pages or print versions. It also helps against duplicate content where a canonical URL is not enough.

It is essential for a test environment, or staging. If a test copy of the site gets indexed, it competes with the live version and may show unfinished content. Password protection is the safest option for staging, with noindex as a second safeguard.

On our site, for example, the 404 error pages carry noindex so they do not appear in results. Pages with noindex are also left out of the sitemap, which should contain only the URLs we want indexed.

What to watch out for

The most common and most expensive mistake is a noindex left on the live site after launch, for example carried over from staging. The site then gradually disappears from search and traffic drops to zero. After every launch or major change it is worth checking that the main pages do not carry the instruction.

The second mistake is using noindex together with a canonical URL pointing to another page. The instructions contradict each other: the canonical says the content is a copy of another page, noindex says the page should not be indexed at all. The search engine may then ignore both signals.