How does Retrieval Augmented Generation (RAG) help LLMs overcome knowledge limitations?

RAG enhances an LLM's prompt with relevant information retrieved from an external knowledge base. This allows the model to answer queries based on up-to-date and specific data, beyond its initial training cutoff.

What is 'document chunking' and why is it important for RAG?

Document chunking involves splitting large documents into smaller, manageable 'chunks' or segments. This is crucial for RAG because LLMs have limited context windows and perform worse with larger inputs, making smaller, relevant chunks more efficient for retrieval.

What is the difference between semantic search and keyword-based search (like BM25) in RAG?

Semantic search retrieves documents based on meaning similarity, even if exact keywords aren't present. Keyword-based search, like BM25, focuses on exact word matches. Often, a hybrid approach combining both yields better results for natural language processing tasks.

How do AI agents execute actions in real-time?

AI agents use 'tools' which are functions or APIs connected to real-time systems or external services. The LLM predicts which tool to use, executes it, and then receives the tool's output to formulate a final response, enabling dynamic interactions.

What are some common types of tools that AI agents can use?

AI agents can use tools for web search, querying databases, performing calculations (e.g., a calculator), executing code (like plotting graphs), and even initiating actions like sending emails. These extend the LLM's capabilities beyond text generation.

What is the problem of tool conflicts and how is 'tool selection' used to mitigate it?

Tool conflicts occur when multiple tools have overlapping capabilities, leading the LLM to choose an unintended tool. Tool selection is a process that filters available tools based on user context, past interactions, or user profiles to present only the most relevant subset to the LLM.

What is the purpose of the MCP (Modular Capabilities Protocol) for AI agents?

The MCP aims to standardize how tools are defined and integrated across different LLM providers. It simplifies tool development and allows tools to be universally compatible with any LLM client that supports the protocol.

How does prompt caching reduce the cost of LLM interactions, especially with tools?

Prompt caching stores the fixed parts of a prompt (like tool definitions and system instructions). If these fixed parts are reused across multiple queries, the user only pays for their processing once, significantly reducing costs compared to re-processing them every time.

Explain the ReAct (Reasoning and Acting) framework for AI agents.

ReAct is an agent framework where the LLM follows a loop of 'Observe', 'Think', 'Act', and 'Observe' again. It starts by observing the current state, plans an action, executes it using a tool, and then observes the new state, iterating until the goal is achieved.

How do programming agents differ from general LLMs in handling context and information retrieval?

Programming agents often prioritize exact match retrieval using tools like 'grep' over traditional semantic RAG, as code often requires precise matches. They also benefit from 'agents.md' files to define persistent rules and preferences, and 'skills' to encapsulate repeated workflows.

Key Moments

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 6 - AI Agents

Stanford Online

Education6 min read101 min video

Nov 13, 2026|15,951 views|225|2

Stanford Stanford Online LLM Large Language Models

Save to Pod

Key Moments

TL;DR

AI agents can now perform complex tasks, but their reliance on specific prompts and the potential for over-reliance on tools like "grep" in programming raise concerns about robustness and security.

Key Insights

Retrieval-Augmented Generation (RAG) enhances LLMs by retrieving relevant information, addressing the knowledge cutoff and limited context window of base models.

Bi-encoders and cross-encoders are used in RAG for candidate retrieval and re-ranking, respectively, with BM25 offering a keyword-matching alternative.

Tool calling allows LLMs to execute actions beyond text generation by interfacing with external functions, categorized into three main types: information retrieval, computation, and code execution.

The ReAct framework combines reasoning and action, enabling agents to observe, plan, act, and observe again in a loop to achieve goals.

Agent frameworks like 'harness' provide the infrastructure around an LLM, including tools, orchestration, compression strategies, and memory management.

Multi-agent systems, including agent swarms, allow multiple AI agents to collaborate and communicate, presenting both opportunities for complex problem-solving and security risks.

Bridging LLM limitations with retrieval-augmented generation

Large Language Models (LLMs) have limitations, primarily their knowledge being restricted to their training data, leading to a 'knowledge cutoff' date. For instance, GPT-6 Astra's cutoff is April 30, 2026, making it unaware of events after that date. Furthermore, LLMs cannot perform actions; they can only respond to prompts. Retrieval-Augmented Generation (RAG) addresses these issues by enhancing prompts with relevant external information. The RAG process involves retrieving relevant documents, augmenting the prompt with this retrieved content, and then generating a response. This approach circumvents the limitations of a fixed context window (even a large one like GPT-6 Astra's million tokens) and the performance degradation seen with longer inputs (the 'needle in a haystack' problem). RAG is crucial for enabling LLMs to access and utilize up-to-date or specific information not present in their training data.

The mechanics of retrieval in RAG systems

Effective retrieval is the cornerstone of RAG. Documents are first processed into smaller 'chunks' and then represented as embedding vectors. The retrieval process typically involves two stages: candidate retrieval and re-ranking. Candidate retrieval aims to maximize recall by quickly scanning the knowledge base for potentially relevant documents, often using computationally inexpensive methods like bi-encoders (which encode query and document separately) or BM25 (for keyword matching). Bi-encoders calculate similarity scores between query and document embeddings, often leveraging approximate nearest neighbor algorithms for efficiency. The transcript highlights that semantic similarity from bi-encoders might not always be sufficient, leading to the consideration of keyword-based methods like BM25. Hybrid approaches combining both semantic and keyword search often yield better results. Candidate retrieval might return hundreds of potential documents.

Refining relevance through re-ranking and evaluation

Following candidate retrieval, a more sophisticated 're-ranking' stage refines the results to capture finer nuances. This stage is more computationally intensive but operates on a smaller set of documents (e.g., 100 candidates). Cross-encoders are often used here, processing the query and document together in a single model to better understand their relationship. Metrics like Normalized Discounted Cumulative Gain (NDCG), Mean Reciprocal Rank (MRR), precision, and recall are used to evaluate the effectiveness of the retrieval process, assessing how well the top-ranked documents align with actual relevance.

Enabling LLMs to take action with tool calling

Tool calling extends LLMs beyond text generation by allowing them to interact with external functionalities. Tools can be broadly categorized into information retrieval (e.g., searching databases, web browsing), computation (e.g., calculators for precise math), and code execution (e.g., generating plots, running scripts). When a user prompts an LLM with a request that requires an action, the LLM predicts which tool to use and what arguments to pass. This involves a three-stage process: 1) The LLM predicts the required action using available tools. 2) The predicted action is executed by the backend system. 3) The results from the backend are fed back to the LLM to formulate a final, natural language response. For instance, to find a teddy bear nearby, an LLM might first call a tool to get the user's location, then call another tool to query a backend system for nearby teddy bears, and finally, present the findings.

Standardizing tool integration with protocols like MCP

To manage the complexity of integrating numerous tools with various LLMs, standardization protocols like the Model Communication Protocol (MCP) are essential. MCP defines a 'server' that hosts tool implementations and a 'client' that interacts with these servers. It provides unified templates for defining tools, their arguments, and outputs. It also defines 'prompts' for specific workflows and 'resources' that applications can provide to tools. This standardization simplifies tool development and integration, allowing a tool to be used across different LLM providers that support the protocol. Prompt caching is a crucial cost-saving mechanism, especially when tool definitions are included in the prompt, reducing costs significantly.

Orchestrating LLM behavior with the ReAct framework

The ReAct (Reasoning and Acting) framework provides a structured approach for LLMs to interact with tools and achieve goals. It involves a loop of observing the current state, planning actions, executing those actions ('act'), and then observing the outcome again. This iterative process allows the LLM to dynamically adjust its strategy based on the feedback from tool execution. For example, if a teddy bear is cold, the LLM might observe this, plan to check the temperature, act by calling a temperature-sensing tool, observe the current temperature, and then plan to adjust the thermostat if necessary. This framework is often learned through supervised fine-tuning.

Managing context and memory for long-term agent interaction

LLMs have a finite context window, which poses a challenge for maintaining long conversations or complex workflows. 'Compression strategies' are employed to reduce the amount of information in the context window, such as summarizing previous turns or removing information no longer relevant in a sliding window approach. Beyond short-term memory (the context window), 'long-term memory' allows agents to recall information over extended periods. This can be achieved through RAG on past conversations or by using dedicated memory tools to store and retrieve specific facts or user preferences. Proper memory management is crucial for agents to retain context and user intent over time.

The rise of programming agents and multi-agent systems

Programming agents, a prominent application of AI agents, leverage tools like 'grep' for efficient code retrieval due to its precision over semantic similarity. They operate within execution environments and often employ automated approval mechanisms for actions. Concepts like 'agents.md' provide configuration rules, while 'skills' encapsulate repeatable workflows. The broader ecosystem includes frameworks like 'Pi' for streamlined agent development. Furthermore, multi-agent systems, including agent swarms, enable collaboration among multiple agents. While powerful for complex tasks, they introduce challenges like conflicting tool scopes ('benign' vs. 'malicious' conflicts) and potential security vulnerabilities, as demonstrated by the Hugging Face incident. Tool selection mechanisms help mitigate conflicts by choosing the most relevant subset of tools for a given query. The complexity of these systems is growing, with ongoing research into areas like computer control and more sophisticated multi-agent coordination.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Concepts

OpenAI Prompt Caching Cost Savings

Data extracted from this episode

Type	Cost per Million Tokens	Relative Saving Factor
Regular Input	10 USD	1x
Cached Input	1 USD	10x

Common Questions

Traditional LLMs are limited by their training data knowledge cutoff, meaning they cannot access real-time information. Additionally, they can only answer questions and cannot perform actions or execute procedures.

Topics

Ai Agents Tool Use AI & Machine Learning Programming & Software Large Language Models Prompt Engineering Context Management Multi-Agent Systems Agent Architectures

Mentioned in this video

Concepts

CME 295

The course code for the lecture, focusing on Transformers and LLMs.

Retrieval-Augmented Generation

A technique to enhance LLM prompts with relevant retrieved information to overcome knowledge limitations.

Needle in a Haystack

A test used to measure how LLM performance degrades with larger input sizes.

Virtual Document Embedding

A method to bridge the gap between query and document embedding nature by generating a dummy document from the query.

BM25

A keyword-based retrieval metric that measures the similarity between a query and a document by comparing exact words.

React

A framework adopted in 2023 that combines "Thought" and "Action" to guide LLMs in achieving goals through observation, planning, acting, and subsequent observation.

A2A

A standardization method for agents released last year, defining skill sets and functions for inter-agent communication.

Software & Apps

ChatGPT

A large language model released in late 2022, used as an example for early programming assistance.

BERT

An encoder-only transformer model mentioned as a suitable architecture for re-rankers due to its bidirectional attention mechanism.

Python

A programming language commonly used by LLM providers to define and express tools for agents.

Google Drive

A cloud storage service mentioned as an example of a tool that could retrieve documents, highlighting potential tool conflicts.

Companies

OpenAI

Provider of LLM models, whose pricing structure for cached vs. uncached prompts is used as an example of cost savings.

Hugging Face

A platform mentioned in the context of agent swarms and potential vulnerabilities, specifically a past breach.

Products

Raspberry Pi

A programming agent framework praised for its simplicity and minimal instructions, often used for comparing models within a single programming framework.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free