What Is RAG (Retrieval-Augmented Generation)?

A model's training data has a cutoff date and never included your private documents. Ask it about last week's incident report and it either declines or guesses. RAG handles this at inference time instead of retraining: search a knowledge base for text relevant to the question, then paste that text into the prompt before the model answers.

It also cuts hallucination on factual questions, though it does not remove it. The model is now working from supplied text rather than from memory, which is a better position, not a guarantee.

Documents are split into chunks and converted into vector embeddings, stored in a vector database. At query time the question is embedded too, and the database returns the chunks whose embeddings sit closest to it. Those chunks go into the prompt, and the model answers from that retrieved text.

Chunk size, embedding model, and how many chunks to retrieve are the knobs, and each one trades against the others. Bigger chunks carry more context per hit but blur similarity matching. More chunks give the model more to work with but cost tokens and can bury the one passage that answers the question.

Similarity search finds text that sounds like the query, not text that answers it. Ask which tool doesn't support a feature and retrieval hands back chunks about the tools that do, because those are the closest match to the words in the question.

It also can't reason across many documents at once unless retrieval happens to surface all of them together. The bigger the knowledge base, the less often that happens.