Indexing

What it means

Indexing is the process in which a search engine finds a page, downloads it, processes its content and stores it in its database, the index. Only pages in the index can appear in search results. A page that is not indexed does not exist for search, however good its content.

The process has two stages. First the search engine’s crawler discovers and crawls the page, typically through links from other pages or the XML sitemap. Then it judges whether the page deserves a place in the index: whether it is unique, useful and technically accessible.

A search engine does not have to index everything it finds. Pages with duplicate or very thin content, pages blocked in robots.txt or marked noindex, and pages whose canonical URL points elsewhere will not be in the index.

How it works in practice

For a new website or section, we recommend checking in the search engine’s webmaster tools how many pages are indexed and why some are not. The usual causes are a noindex left over from the development version, faulty redirects or duplicates.

On a multilingual site every language version must be indexed and linked through hreflang. Glossary pages like this one are generated statically so that search engines load them quickly and in full, without waiting for scripts.

Loading speed and technical quality, also measured by Core Web Vitals, affect how often and how deeply a search engine crawls a site. A slow site with thousands of pages may be indexed only in part.

What to watch out for

The most common and most expensive mistake is launching a new site with the noindex from the test version still in place. The company waits weeks for traffic that never comes, and the old organic traffic is lost.

The second risk is trying to index everything, including filters, internal search and empty pages. The search engine wastes its time on worthless URLs and may overlook important pages. Fewer, better pages in the index work better.