Guide

What RAG is

RAG (retrieval-augmented generation) works like this: when a question comes in, the relevant passage is found in your own material first, and the answer is written from it. So the model can answer from material it never learned.

In short

Find it, read it, answer

A language model answers from what it learned. It does not know the staff handbook, last month's contract or your internal guidelines, because none of that was public.

RAG adds a lookup step between the question and the answer. The model reads the passage that was found and writes the answer.

That is what makes it possible to show which document and which passage the answer came from, next to the answer itself. Whoever receives it can check.

What runs

From loading the material to getting an answer

The first two steps run once when material is loaded. The last two run on every question.

Split it

Documents are cut into passages. Too large and unrelated content gets mixed in; too small and the thread is lost.

Turn it into numbers

Each passage is stored as an embedding, a block of numbers. Text with a similar meaning ends up with similar numbers.

Find it

The question is turned into numbers the same way and the closest passages are pulled out. Some setups then narrow them down and reorder them once more.

Write the answer

The chosen passages go to the model together with the question, and the model writes the answer. The passages it read can be shown next to the answer.

Side by side

How this differs from fine-tuning

Fine-tuning trains the model further on your material. RAG fits the work where the material keeps changing, and the two are often used together.

When the material changes

Fine-tuning has to be run again every time the material changes. With RAG the changed document is simply loaded again.

When you need the source

A fine-tuned model cannot say where an answer came from. RAG points at the passage it read.

Limits

Where it falls down

Using RAG does not settle everything. Most of the trouble comes from these three.

A wrong lookup gives a wrong answer. Pick the wrong passage in the lookup step and the model writes a wrong answer from it.
How the material is split changes the result a lot. In a document full of tables and figures, cutting out only the text throws the meaning away.
Even when it finds the right passage, the model can misread it. A person makes the final check.

Common questions

Does our material leave our hands?

It depends on where the model runs. With a model run by another company, the passages found for each question are sent to that company. Run everything on your own servers and nothing leaves. Which way to go is settled before you start.

Is a small amount of material enough?

It runs on a handful of documents. With few candidates to choose from, though, unrelated passages surface easily, so it helps to keep the scope of the questions narrow.

Try it on your own material

With LabRAG you load the material, describe what you need in plain words, and the service that does that work gets built.

See LabRAG hello@labgl.net