AI hallucination

What it means

An AI hallucination is when a language model confidently states false information, such as an invented section of law, figure, court ruling or source. The answer sounds convincing and grammatically correct, so the error is hard to spot without checking.

The cause lies in how the model works: it predicts the most likely continuation of text based on patterns in its training data rather than looking up verified facts. When it lacks information, it often produces an answer that merely looks true instead of admitting uncertainty.

The risk is greatest for specialist topics, precise figures, recent legal changes and local rules, such as Slovak tax rates. In tax and legal matters especially, output from generative AI must be checked against the legislation or with a professional.

How it works in practice

Connecting the model to verified sources, such as internal documents or current legislation, and requiring answers to cite sources reduces hallucinations. The references must still be checked, because the model sometimes invents citations too.

Companies deploying a customer chatbot should restrict its answers to verified content and hand over to a person when uncertain. The company, as operator, may be liable for wrong information the chatbot gives a customer.

For internal use of AI, a clear rule helps: AI may prepare a draft, summary or translation, but anything going to a client, an authority or into the accounts must be checked by a person. With AI agents that act autonomously, oversight matters even more.

What to watch out for

There are known cases worldwide of lawyers filing court submissions containing AI-invented judgments and being sanctioned. The same risk applies to a tax return or contract built on a non-existent provision, where the consequences fall on the signatory, not the tool.

The AI Act places obligations mainly on providers and deployers of high-risk systems and requires transparency when people interact with AI. Responsibility for the correctness of decisions nevertheless remains with the company that uses the output.