AI Explained
RAG: retrieval-augmented generation
Models only know what they were trained on. RAG looks up the right facts from your own documents first, then has the model answer from them.
A language model only knows what was in its training data. It doesn't know your company's docs, last week's policy change, or the contents of your support tickets. Ask anyway and it may guess. RAG, retrieval-augmented generation, fixes that by looking things up first.
The idea in one line
Before the model answers, retrieve the most relevant pieces of your own content, add them to the prompt, and let the model generate an answer from them.
The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, which combined a language model with a searchable index of documents so answers could draw on knowledge the model itself didn't store.
How a RAG pipeline works
Once, ahead of time (indexing):
- Split your documents into chunks, a few paragraphs each.
- Turn each chunk into an embedding, a vector that captures its meaning.
- Store the vectors in a vector database or search index.
Every question (retrieval and generation):
- Embed the user's question.
- Find the chunks whose vectors are closest to it.
- Put those chunks into the prompt with an instruction like "Answer using only the context below, and say if it isn't there."
- The model answers, ideally citing which chunks it used.
Why it's so popular
- Fresh knowledge. Update a document and the next answer uses it. No retraining.
- Your private data. The model can answer from content it never saw in training.
- Fewer hallucinations. Answers are grounded in real text, and you can show the sources.
- Cheaper than fine-tuning for most "know our stuff" problems.
Where RAG goes wrong
Most RAG bugs are retrieval bugs. If the right chunk never comes back, the model can't use it. The usual fixes:
- better chunking (keep related text together),
- hybrid search (keywords plus embeddings),
- a reranker to reorder the top results,
- and evals that check whether the right chunks were found, not just whether the answer sounds good.
Look it up, then answer.