Web crawler

What it means

A web crawler is a search engine's or AI service's program that automatically follows links across the web, downloads pages and passes them on for processing. Only after this step can a page go through indexing and appear in search results.

Well-known crawlers include Googlebot and Bingbot, and they have been joined by language model crawlers that collect text for training or for real-time answers. Each identifies itself with its own user agent, and well-behaved crawlers follow the rules in robots.txt.

A crawler does not read a site like a person. It sees HTML, links, headings and structured data, and content that appears only after JavaScript runs is processed later or not at all. That is why it helps SEO when the main text is already in the server-delivered HTML, for example through SSR.

How it works in practice

Robots.txt decides which parts of a site a crawler may visit, while the noindex tag controls what goes into the index. Crawlers find new pages through links and the XML sitemap, whose address is worth listing directly in robots.txt.

On large sites the key question is how many pages the crawler gets through, which is the crawl budget. A slow server, errors and endless filters mean important pages are visited rarely and price changes or new products show up in search with a delay.

For AI crawlers an additional file, llms.txt, offers a short overview of the site. A firm can decide which AI crawlers to allow: blocking protects content from training but lowers the chance that AI assistants will mention the firm and link to its pages.

What to watch out for

A user agent is easy to fake, so not every visitor calling itself Googlebot is really Google. A crawler is verified by a reverse DNS lookup or the published IP ranges, which matters when configuring a firewall and protection against malicious bots.

Robots.txt is not a security tool but a polite request that malicious crawlers ignore. Non-public data must be protected by a password or login, not by a robots.txt rule, which actually reveals where the sensitive content is located.