Key Moments
RAG Explained: From BM25 to Agentic RAG | Stanford CS329Z Lecture 3 with Diyi Yang
Stanford OnlineKey Moments
Retrieval-Augmented Generation (RAG) can improve LLM responses by providing external knowledge, but its effectiveness hinges on precise chunking and retrieval, with complex agentic RAG systems still prone to cascading errors.
Key Insights
RAG aims to augment LLMs by retrieving relevant information rather than having models memorize everything, offering an interpretable process with verifiable sources.
Classic RAG employs a retriever-reader framework, where initial document retrieval narrows down millions of articles to a few, which a reader model then processes for a precise answer.
DPR (Dense Passage Retrieval) models, trained with contrastive loss, can outperform traditional methods like BM25, especially when trained on as few as 1000 labeled question-answer pairs.
Advanced RAG techniques include query rewriting, reordering, summarization, and modular approaches that break down the pipeline into searchable, routable, and writable components.
Agentic RAG systems, like Self-RAG and C-RAG, introduce mechanisms for retrieval at decision points, self-critique, and confidence scoring, moving towards more autonomous AI agents.
The effectiveness of RAG can be impacted by errors in retrieval (e.g., outdated embeddings, incorrect documents), reader errors (e.g., 'lost in the middle', conflicting evidence), and prompt injection attacks.
The core idea of RAG: augmenting instead of memorizing
Retrieval-Augmented Generation (RAG) addresses the limitations of Large Language Models (LLMs) by providing them with external, up-to-date information at the time of query, rather than relying solely on their training data. This dynamic process allows LLMs to access and incorporate relevant knowledge, leading to more accurate and verifiable responses. A key benefit highlighted is interpretability, as RAG systems can cite their sources, enabling users to cross-reference information. Unlike traditional LLMs that might 'hallucinate' or provide outdated information, RAG aims to ground responses in factual evidence. The process is also dynamic, meaning the knowledge base can be updated without retraining the model's weights, which is crucial for rapidly changing information.
The classic retriever-reader framework
The foundational RAG architecture, often called the retriever-reader framework, consists of two main components. First, a retriever scans a vast corpus of documents (potentially millions) to identify a small set of highly relevant passages based on a given query. This significantly narrows down the search space. Second, a reader model then processes these retrieved passages to extract or generate a precise answer. Early implementations used TF-IDF for retrieval, while neural models were employed for reading. Modern approaches can leverage powerful LLMs for the reader component, performing zero-shot reading capabilities. The effectiveness of this pipeline depends on both efficient retrieval and accurate reading, aiming to synthesize information from the retrieved context.
Advancements in retrieval: from BM25 to Dense Passage Retrieval (DPR)
The evolution of retrieval methods has been central to RAG's progress. Traditional methods like BM25, a term-matching approach rooted in TF-IDF, remain a strong baseline due to their speed and effectiveness in capturing exact term matches. However, it struggles with semantic nuances and paraphrasing. Dense Passage Retrieval (DPR) emerged as a powerful neural alternative. DPR uses separate BERT encoders for queries and passages to generate dense vector representations. These vectors are then used for efficient similarity searches, typically via Maximum Inner Product Search (MIPS). Crucially, DPR models can be trained using contrastive loss, which encourages positive (relevant) query-passage pairs to be closer in the embedding space and penalizes negative pairs. Research shows that even with a modest number of labeled examples (around 1000 pairs), DPR can significantly outperform BM25, demonstrating the power of learned semantic representations.
Enhancing RAG with sophisticated indexing and chunking strategies
Effective RAG relies heavily on how documents are pre-processed and indexed. Chunking involves dividing long documents into smaller, manageable pieces for retrieval. Simple fixed-size chunking can be improved by incorporating overlap to maintain context across boundaries. More advanced semantic chunking uses natural breaks like sentences or paragraphs. Context-rich chunking adds metadata or summaries, while AI-driven dynamic chunking uses LLMs to identify semantic breakpoints. Indexing techniques are also critical for fast retrieval. Vector databases are common, employing Approximate Nearest Neighbor (ANN) algorithms like Hierarchical Navigable Small Worlds (HNSW) to balance search speed with accuracy. Methods like 'late chunking' process documents first to capture document-level semantics before splitting, yielding richer representations, especially for smaller chunks. This contrasts with 'early chunking' where documents are split first. Hybrid approaches, combining sparse (BM25) and dense (DPR) retrieval, aim to leverage the strengths of both, capturing both semantic meaning and exact term matches.
Agentic RAG: empowering LLMs with decision-making
The latest frontier in RAG involves creating 'agentic' systems where the LLM actively decides when and how to retrieve information. This moves beyond a passive retrieval step. Systems like Self-RAG train a single model to perform retrieval, critique, and generation. It retrieves evidence on demand, generates responses segment by segment, and critically evaluates each segment for relevance and support. C-RAG (Critique-RAG) uses a separate evaluator model between the retriever and generator to assess retrieved documents' relevance and confidence, potentially re-querying or re-ranking if confidence is low. Search R1 employs Reinforcement Learning (RL) to learn an optimal search strategy, allowing the agent to iteratively refine queries and search multiple times to gather information, mimicking a human-like research process. These agentic approaches aim to improve robustness and reduce errors by embedding decision-making capabilities directly into the RAG pipeline.
Challenges and limitations in RAG systems
Despite advancements, RAG systems face several challenges. Retrieval failures can occur if the correct evidence isn't found, embeddings are outdated, or the search mechanism is flawed. Reader errors, such as the 'lost in the middle' phenomenon (where LLMs struggle to utilize information presented in the middle of long contexts) or conflicting retrieved evidence, can degrade response quality. Furthermore, RAG pipelines are susceptible to cascading errors, where an initial mistake in retrieval amplifies through subsequent steps. Prompt injection attacks, where malicious input manipulates the RAG system's behavior, also pose a significant security risk. Addressing these limitations is crucial for building reliable and trustworthy RAG-powered AI agents.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Studies Cited
●Concepts
Common Questions
RAG is a technique that enhances large language models by allowing them to retrieve relevant information from an external knowledge source before generating a response. This provides a factual basis for answers and allows for dynamic updates without retraining the model.
Topics
Mentioned in this video
A dataset used for evaluating RAG performance, originating from Google searches.
A dataset (Stanford Question Answering Dataset) used for training neural models for question answering, relevant to RAG's reader component.
A dataset used for evaluating RAG performance, consisting of trivia questions.
A neural network-based approach for retrieving dense vector representations of passages, often outperforming traditional methods like BM25.
An efficient approximate nearest neighbor search algorithm used for vector indexing in RAG, offering a hierarchical structure for fast retrieval.
A recent large language model that supports general internet search capabilities, enabling RAG-like functionality for real-time information.
A late interaction model that achieves high performance by computing token-level similarities between queries and documents.
Models that are the core of the systems discussed, used for generating text and understanding information, and also as potential judges for evaluation.
A classic and widely used text retrieval algorithm based on TF-IDF, often used as a baseline and for hard negative sampling in RAG.
An example of a large language model that can be used for zero-shot learning and as a reader function in older RAG frameworks.
A model that provides embeddings for both text and images, enabling multimodal RAG.
An older encoder-decoder architecture used as an example for the generator component in RAG, demonstrating sequence-to-sequence modeling.
A transformer-based language model commonly used in early RAG systems for encoding queries and passages, though now considered historical.
A technique that enhances large language models by retrieving relevant information from an external knowledge base before generating an answer.
An agentic RAG approach that uses a separate evaluator between the retriever and generator to assess retrieved documents before generation.
An agentic RAG approach that uses reinforcement learning to learn a search process, allowing the agent to iteratively refine queries and search.
An agentic RAG approach where a single model is trained to retrieve, critique, and generate text, incorporating self-reflection.
A security vulnerability where malicious input is injected into a prompt to manipulate an LLM or RAG system's behavior.
More from Stanford Online
View all 174 summaries
64 minStanford CS193T Thinking with AI | Autumn 2026 | Managing Sources
78 minLLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan
77 minStanford CS193T Thinking with AI | Autumn 2026 | Core Prompting
73 minStanford CS193T Thinking with AI | Autumn 2026 | Introduction to AI
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free