Robots.txt

What it means

Robots.txt is a simple text file in a website's root directory, for example at domena.sk/robots.txt. It contains instructions for the automated robots that crawl the site, both search engine crawlers and AI service crawlers. The rules say which parts of the site may and may not be crawled and where the sitemap is.

The file consists of groups that begin by naming a robot and continue with Disallow and Allow rules for specific paths. Rules can apply to all robots at once or only to selected ones, such as a particular search engine or AI service. The format is standardised and most well-behaved robots respect it.

Robots.txt controls crawling, not indexing. If other sites link to a disallowed page, a search engine may list it in results without reading its content. To keep a page out of results you use the noindex tag, but the robot will only see it if you allow the page to be crawled.

How it works in practice

For an ordinary company website a simple robots.txt that disallows nothing and points to the sitemap is enough. Disallowing makes sense for technical areas such as the admin section, internal search results, the shopping cart or endless filter combinations in an online store, which would waste crawler capacity and add no value.

In recent years robots.txt has also been used to decide whether to open content to AI service crawlers. A company can allow them so that its content appears in assistants' answers, which ties in with GEO optimisation, or block them. An llms.txt file offering AI tools an overview of the content can complement it.

After every site change we recommend checking that robots.txt is not accidentally blocking important pages or the CSS and JavaScript files a search engine needs to render the page. Webmaster tools let you test how the search engine reads the rules and flag errors.

What to watch out for

The most common and most expensive mistake is leaving in a rule that blocks the whole site after launching a new version. Developers add it on the staging environment and it survives deployment. The site then gradually disappears from search. Checking robots.txt should be part of every launch.

Robots.txt is not a security tool. The file is public, so anyone can read which paths you want to hide, and malicious robots ignore it anyway. Sensitive content must be protected by login or not be on the public website at all, not merely disallowed in robots.txt.