Duplicate content

What it means

Duplicate content is the same or very similar text available at several URLs. It can arise within one site, for example when an online store lists a product in three categories under three addresses, or across sites, such as when dozens of stores use the same manufacturer description.

Search engines do not want to show the same thing several times, so they pick one of the duplicate versions and suppress the rest. The problem is that they may not pick the one the site owner wants, and the strength of links and signals is split across several addresses instead of reinforcing one.

Duplicate content is not a reason for a penalty unless it involves deliberately copying others’ content or generating thousands of near-identical pages. It is more a technical problem that reduces a site’s effectiveness and wastes the time search engines spend crawling it.

How it works in practice

The most common sources of duplicates are www and non-www versions, trailing slashes, URL parameters from filters and campaigns, pagination, print versions or a publicly accessible test environment. They are fixed with a canonical URL, a 301 redirect to the main version or a noindex tag for pages that need not appear in results.

Online stores should write their own descriptions at least for their best-selling products. The whole market has the manufacturer’s text, and a search engine has no reason to prefer your store. Multilingual sites need a correct hreflang setup so that language versions are not treated as duplicates.

On our site each term has one canonical address in each language, and the language versions are linked through hreflang. That is also why each term’s explanation is written individually rather than assembled from shared paragraphs repeated across hundreds of pages.

What to watch out for

A common mistake is handling duplicates with robots.txt. If crawling is blocked, the search engine cannot see the canonical tag either and may index the page from links. Duplicates call for a canonical URL or a redirect, not blocking.

Watch template-generated text as well, such as hundreds of “accounting in city X” pages where only the city name changes. Search engines treat them as duplicate or thin content, and at scale this can harm the whole site, not just those pages.