PATTERN
What Is RAG (Retrieval-Augmented Generation)?
RAG (Retrieval-Augmented Generation) is a pattern where a system pulls relevant documents or passages at query time and drops them into the model's context before it answers, instead of relying only on what the model learned in training.
What the video explains
Why the app has to look things up
RAG stands for Retrieval Augmented Generation. Say a user asks a question: what is an LLM? The question goes into an app, the RAG app, which must send back an answer. But the model alone was never trained on your documents, so the app has to look up your knowledge first.
Indexing: preparing the knowledge once
Before any question arrives, the knowledge is prepared once, ahead of time. This is called indexing. It starts with your raw documents: articles and data about LLMs and AI, or anything you want the app to know. A long document is too big to search well, so each one is split into chunks, which are small pieces a few paragraphs long.
Next, each chunk is turned into a vector, a list of numbers that captures the meaning of the text. The vectors are stored in a vector database. That was indexing, and it happens once, before any question.
Retrieval: finding the closest chunks
When the question arrives, it is turned into a vector too, so it can be compared with the database. The app then searches by semantic proximity, which means closeness in meaning, not matching keywords. Out of everything in the database, it keeps only the best few chunks, ranked with the closest one first.
Augmented generation: a cited answer
Now the app builds a prompt that puts together the question and the best chunks from the search. This prompt is the context the model answers from. The LLM reads that context, understands it, writes an answer, and cites where each piece of information came from.
The answer goes back to the app and to the user, grounded in real documents with a source to prove it, not a guess. In short, indexing prepares the knowledge once, retrieval finds the best chunks for each question, and augmented generation writes the answer from them.
Why retrieval instead of just asking the model
A model's training data has a cutoff date and never included your private documents. Ask it about last week's incident report and it either declines or guesses. RAG handles this at inference time instead of retraining: search a knowledge base for text relevant to the question, then paste that text into the prompt before the model answers.
It also cuts hallucination on factual questions, though it does not remove it. The model is now working from supplied text rather than from memory, which is a better position, not a guarantee.
The pipeline
Documents are split into chunks and converted into vector embeddings, stored in a vector database. At query time the question is embedded too, and the database returns the chunks whose embeddings sit closest to it. Those chunks go into the prompt, and the model answers from that retrieved text.
Chunk size, embedding model, and how many chunks to retrieve are the knobs, and each one trades against the others. Bigger chunks carry more context per hit but blur similarity matching. More chunks give the model more to work with but cost tokens and can bury the one passage that answers the question.
Where RAG breaks
Similarity search finds text that sounds like the query, not text that answers it. Ask which tool doesn't support a feature and retrieval hands back chunks about the tools that do, because those are the closest match to the words in the question.
It also can't reason across many documents at once unless retrieval happens to surface all of them together. The bigger the knowledge base, the less often that happens.
FAQ
- Is RAG the same as fine-tuning?
- No. Fine-tuning changes the model's weights; RAG changes what's in the prompt at inference time. They solve different problems and can be combined.
- Do I need a vector database to do RAG?
- Not strictly. Any retrieval method that finds relevant text works: keyword search, a SQL query, or a vector database. Vector search is popular because it captures semantic similarity, not just keyword overlap.