Retrieval-augmented generation

What it means

RAG, retrieval-augmented generation, is a technique in which a language model first retrieves relevant passages from a company's trusted documents and only then builds its answer from them. The model thus answers not just from what it learned in training but from current, verified sources.

The main benefits are fewer hallucinations and the ability to work with internal data the model has never seen: policies, contracts, price lists or a database of client questions. Answers can include references to the source so the user can check them.

Technically, documents are split into smaller chunks, converted into numerical vectors and stored in a search index. When a question comes in, the system finds the most similar chunks and inserts them into the prompt with an instruction to answer only on that basis.

How it works in practice

Typical uses are an internal assistant answering staff questions about procedures, or a website chatbot answering clients from current terms and prices. An accounting firm can use it to open up an internal knowledge base on closing or VAT procedures.

Quality depends on the source material. Outdated, contradictory or poorly structured documents produce poor answers even with an excellent model. Before launch it is important to clean up the knowledge base and decide who keeps it current.

The system should be tested on real questions, measuring whether answers are correct and grounded in the right sources. On specialist topics it is sensible for a person to review answers to clients until the system has proven reliable.

What to watch out for

RAG reduces hallucinations but does not eliminate them. The model can misread a retrieved passage, combine unrelated chunks or answer even when no relevant source was found. Instructing it to admit when it does not know helps.

Every document loaded ends up in the search index. If personal data or confidential contracts are among them, the system may show them to an unauthorised user. Document access rights must therefore be reflected in retrieval as well.