Skip to main content

Introduction to RAG

TL;DR
  • RAG = retrieve relevant context at query time, then let the LLM answer from it.
  • Fixes a bare LLM's three limits: stale knowledge, no private data, hallucination.
  • Two phases: indexing (offline) and querying (per request).

This section establishes what RAG is and the problem it solves before any code.

What RAG is​

Retrieval-Augmented Generation gives a language model knowledge it wasn't trained on by retrieving relevant text at question time and placing it into the prompt. The model then answers from that supplied context instead of relying only on what's baked into its weights.

A plain LLM is a closed-book exam; RAG turns it into an open-book one — the model still writes the answer, but it can look up the right page first.

Why RAG exists​

It fixes three limits of a bare LLM:

  • Stale knowledge — the model only knows its training data up to a cutoff; it can't see your latest or private documents.
  • No private data — your company wiki, PDFs, and tickets were never in training.
  • Hallucination — asked something it doesn't know, the model confidently invents an answer. Grounding it in retrieved sources curbs this and makes answers citable.

The two phases​

RAG's two phases — indexing once, querying per question

Everything later in the course improves one of these steps — better parsing, chunking, retrieval, or agentic control over the loop.

Cheat sheet​

  • RAG = retrieve relevant context at query time → LLM answers from it.
  • Fixes: stale knowledge, private data, hallucination; enables citations.
  • Two phases: indexing (offline) and querying (per request).
⚠ Common mistakes
  • Thinking RAG "trains" the model — it doesn't; it only adds context to the prompt.
  • Expecting RAG to fix a reasoning gap. RAG fixes a knowledge gap; if the model can't reason over the context, retrieval won't help.
  • Forgetting retrieval can fail — if the right chunk isn't retrieved, the model can't answer (or will hallucinate). Most "RAG bugs" are retrieval bugs.

RAG grounds an LLM in retrieved context. It fixes stale knowledge, private data, and hallucination — and makes answers traceable to a source.

Quick self-check

What three problems does RAG solve?

Stale knowledge (training cutoff), no access to private data, and hallucination — plus it makes answers citable.

What are the two phases of RAG?

Indexing (offline: load → chunk → embed → store) and querying (per question: embed → retrieve → augment prompt → generate).

Does RAG change the model's weights?

No. It only supplies context in the prompt at query time. Changing weights is fine-tuning.

Related: Core Components → · Glossary