Retrieval-Augmented Generation

Retrieve the most relevant external knowledge per query and feed it to the generator — context construction that treats the retriever as an agent tool.

RAG retrieves relevant information from external stores and joins it with the user prompt so a generator answers with grounded context. It is feature engineering for foundation models: give the model the facts it needs for this input. Benefits include denser answers, fewer hallucinations, and per-user data scoping. Without retrieval, models may refuse unknown facts or — worse — hallucinate convincing ungrounded answers. Retrieving irrelevant snippets is not neutral either — see the Chekhov's gun retrieval fallacy for why padding context with loosely-related hits actively hurts answers rather than just wasting space.

Lewis et al. frame RAG as a hybrid memory recipe: parametric memory (pretrained generator weights) plus non-parametric memory (a dense index queried by a retriever). The non-parametric side is why knowledge can be inspected for provenance, expanded without retraining, and replaced when the world changes — problems parametric-only models handle poorly. Both components arrive pretrained (for example BART + DPR over Wikipedia), so knowledge access is present without training memory networks from scratch for each task. Index hot-swapping is the operational form of that claim: rebuild or swap the Wikipedia dump (for example 2016 vs 2018) and leader- identity questions flip to the matched year's answers (~70% correct) while mismatched indices collapse (~4–12%) — world knowledge updated with no generator retraining.

Long context does not obsolete RAG: data volume grows faster than windows, and models often use long prompts poorly (lost in the middle) while every token costs latency. Small knowledge bases may still fit whole in prompt; larger ones need retrieval. Prefer precise hits over volume, per the Chekhov's gun retrieval fallacy above. Pair retrieved material with context grounding so the generator actually uses the passages rather than drifting back to parametric guesses.

Retrieval query construction spans using the raw user text (risky for long noisy requests), asking the model for a search string (query rewriting), or giving the agent a search tool so it decides when to retrieve — the overlap of RAG and function calling. Architecturally RAG is a special case of an LLM agent where the retriever is a tool, and the agentic pattern generalizes further with richer tool inventories; see agent memory tiers for how retrieved material enters short-term context. Classic latent-document formulations choose between RAG-Sequence and RAG-Token marginalization inside a retriever–generator architecture; prompted systems usually approximate that by concatenating top chunks instead. Prefer generation over extractive retrieval when answers need synthesis, clue-bearing passages, or parametric fallback.

Sources

  • AI Engineering (Chip Huyen), ch. 6
  • Prompt Engineering for LLMs (John Berryman), ch. 5
  • RAG (Lewis et al.), Abstract; §1–§2