49  Retrieval-Augmented Generation (RAG)

TipTL;DR

RAG augments a generative model with a retrieval step: relevant documents are fetched from an external store at inference time and added to the context. The model’s weights stay fixed; knowledge lives in the retrieval index, not the parameters.

Depends on: Agent Architectures Overview · Alignment & Instruction Tuning (RAG as a fix for stale, parametric knowledge)

49.1 Why this matters

Alignment & Instruction Tuning already introduced why RAG exists: a pretrained model’s knowledge is frozen at training time and hallucinates confidently when asked about anything newer or more specific than its training data covered. This note goes one level deeper, into the retrieval pipeline itself: how a query actually becomes retrieved text, and the specific failure modes and fixes that pipeline has accumulated.

49.2 The mechanism

A RAG system runs four stages before generation even starts, and the retrieved text is then just concatenated into the model’s context alongside the query.

• Embed: documents (and later, the query) are mapped to dense vectors by an embedding model, the same idea as the contrastive encoders in Self-Supervised Learning or CLIP’s text tower.
• Index: document vectors are stored in a vector database that supports fast nearest-neighbor search over millions of entries.
• Query: the user’s query is embedded with the same model, and the index returns the \(k\) nearest document vectors.
• Rerank: a smaller, more expensive model re-scores the top-\(k\) candidates for relevance, since fast approximate nearest-neighbor search trades some precision for speed.

49.2.1 Chunking and embedding choices

Documents are split into chunks before embedding, since a single vector cannot represent an entire long document precisely. Chunk size is a direct tradeoff: smaller chunks retrieve more precisely but lose surrounding context; larger chunks preserve context but dilute the embedding with irrelevant content. The embedding model itself is usually a contrastively-trained encoder, the same family as SimCLR/MoCo, applied to text instead of images.

49.2.2 Naive RAG vs. advanced RAG

The pipeline above, embed the query once and retrieve once, is naive RAG, and it fails when the query itself is a poor search term for what’s actually needed. Two refinements address this directly.

• HyDE (Gao et al., 2023): instead of embedding the raw query, first ask the language model to generate a hypothetical answer, then embed and search with that hypothetical document. A generated answer is closer, in embedding space, to the real answer documents than the original question is.
• FLARE (Jiang et al., 2023): instead of retrieving once up front, retrieve actively, mid-generation, whenever the model’s own confidence in its next tokens drops, triggering a fresh retrieval query at exactly the point new information is needed.

49.2.3 What to observe

• Every stage of the pipeline (embed, index, query, rerank) is a place quality can be lost; RAG’s overall quality is bounded by its weakest stage, usually retrieval, not generation.
• HyDE and FLARE both attack the same weakness, a naive query is often not what should be searched for, from different angles: rewrite the query, or retrieve at a better time.
• RAG requires no change to the underlying model’s weights, the direct contrast to fine-tuning: knowledge is added by changing what’s in the index, not what’s in the parameters.

WarningPitfall: RAG’s latency is not free

Every retrieval step adds a round-trip to the vector index before generation can proceed; FLARE’s repeated mid-generation retrieval trades that latency for freshness, and is only worth it when generation quality would otherwise suffer from stale context.

49.3 Application & impact

Concept here	What it became	Where you see it today
Embed → index → query → rerank	The standard RAG pipeline	Nearly every production LLM application with private or fresh data
HyDE’s query rewriting	Bridging the query-document embedding gap	Common preprocessing step in RAG pipelines
FLARE’s active retrieval	Retrieval as part of the agent loop, not a one-shot step	Agentic RAG systems

NoteKey takeaway

RAG decouples knowledge from weights by adding a retrieval pipeline in front of generation. The pipeline mechanics, chunking, embedding, and when to retrieve, are where most of RAG’s practical quality is won or lost, not in the generation step itself.

← Back to: Agent Architectures Overview

Gao, L., Ma, X., Lin, J., & Callan, J. (2023). Precise zero-shot dense retrieval without relevance labels. arXiv Preprint arXiv:2212.10496.

Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023). Active retrieval augmented generation. arXiv Preprint arXiv:2305.06983.