Key Moments
LLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan
Stanford OnlineKey Moments
LLM context length is limited by quadratic attention complexity, forcing new linear attention methods that trade long-range recall for efficiency, impacting agent design for complex tasks.
Key Insights
Large Language Models (LLMs) are fundamentally generative, modeling the probability of a sequence of tokens, enabling them to predict and generate text.
Transformer architecture, particularly the decoder-only variant, relies on the attention mechanism, where each token can attend to all previous tokens, enabling parallel processing with O(1) sequential complexity.
Linear attention methods reduce the computational complexity from O(N²) to O(N) with respect to sequence length by altering the attention calculation, but can struggle with long-range dependencies.
The pre-training phase for LLMs involves extensive data filtering and cleaning from sources like the internet, retaining only a small fraction (e.g., 1.4%) of high-quality content for training.
Post-training phases like instruction fine-tuning and Reinforcement Learning from Human Feedback (RLHF) are crucial for aligning LLM behavior with desired outputs, though RLHF faces challenges with human preference consistency and potential biases.
Inference for LLMs involves two stages: 'prefill,' which processes the entire prompt in parallel and is compute-bound, and 'decode,' which generates tokens sequentially and is memory-bound, relying on the KV cache.
Defining language models and their generative nature
The lecture begins by defining language models as systems that calculate the probability of a sequence of tokens (words or sub-word units). This probability is derived from the conditional probability of each token given all preceding tokens. Mathematically, this is represented as a product of probabilities, often computed as a sum of log probabilities for numerical stability. Crucially, LLMs are classified as generative models because they can model the probability of any input sequence, distinguishing them from discriminative models that focus on class separation. This generative capability allows LLMs to predict the next token in a sequence, forming the basis for text generation. The process of sampling from these probability distributions, using strategies like greedy, top-k, or top-p, along with temperature scaling, controls the diversity and randomness of the generated output.
Transformer architecture and the pivotal role of attention
The core architecture discussed is the Transformer, particularly the decoder-only variant prevalent in modern LLMs. At the heart of the Transformer is the attention mechanism, which allows the model to weigh the importance of different tokens in the input sequence when processing a specific token. The lecture details the mechanics of causal attention within a decoder, emphasizing that each token can only attend to preceding tokens. This process involves computing query, key, and value vectors, calculating attention scores, applying a causal mask to prevent looking into the future, scaling, and using a softmax function to obtain attention weights. These weights are then used to create a weighted sum of the value vectors, producing the output of the attention layer. This mechanism is critical because it enables parallel processing of tokens within a sequence, leading to an O(1) sequential complexity for processing the entire sequence during the 'prefill' stage, a significant advantage over recurrent models.
Decoding strategies and controlling output generation
To generate text, LLMs sample from the probability distribution of the next token. Several decoding strategies exist: the greedy approach selects the most probable token, while top-k and top-p sampling consider a subset of the most likely tokens. Temperature scaling is a hyperparameter that controls the randomness of sampling; a higher temperature leads to more diverse outputs, while a lower temperature makes the output more deterministic and focused on high-probability tokens. Beam search is a more computationally intensive strategy that explores multiple potential sequences simultaneously to find a higher-probability output. Each strategy presents trade-offs between output quality, diversity, and computational cost, making the choice dependent on the specific application.
The LLM training pipeline: Pre-training, post-training, and agent specialization
The training process is segmented into pre-training, post-training, and potentially agent-specific fine-tuning. Pre-training involves exposing the model to massive datasets, often scraped from the internet and subjected to rigorous filtering to ensure quality. This process aims to imbue the model with general knowledge and language understanding. Post-training refines the model's capabilities through methods like instruction fine-tuning (supervised fine-tuning on instruction-response pairs) and Reinforcement Learning from Human Feedback (RLHF). RLHF, inspired by models like InstructGPT, uses human preferences to train a reward model, which then guides the LLM to generate more desirable outputs. However, RLHF faces challenges related to the inconsistency and biases of human preferences and the potential for 'sycophantic' model behavior. Agent-specific training might involve creating specialized environments, like coding challenges, to train agents for complex, long-term tasks.
Architectural innovations for handling long contexts: Linear attention
The standard Transformer's O(N²) complexity in attention with respect to sequence length (N) becomes a bottleneck for long contexts. Linear attention mechanisms offer a solution by reducing this complexity to O(N). This is achieved by reformulating the attention calculation, often by changing the order of matrix multiplications or removing the softmax. While linear attention significantly improves efficiency and memory usage, it can sacrifice performance on tasks requiring strong long-range dependencies, as demonstrated by benchmarks like 'needle in a haystack.' Hybrid approaches that combine linear attention for most layers with occasional global attention layers are being explored to balance efficiency and effectiveness.
Inference optimization: Prefill vs. decode and speculative decoding
LLM inference is divided into two phases: 'prefill' and 'decode.' Prefill processes the entire input prompt in parallel, involving significant matrix multiplications and is compute-bound. Decode generates tokens sequentially, relying on the KV cache to store intermediate attention states. Decode is memory-bound, as the KV cache grows with the generated sequence length. Techniques like speculative decoding aim to accelerate inference by using a smaller, faster model to predict a sequence of tokens, which are then verified by the larger model. This can significantly reduce latency, especially in domains like code generation where outputs are more predictable, though it adds complexity to the system.
Context engineering for agents: Structuring information effectively
For AI agents, managing context is paramount due to the need to process long and complex information. Simply dumping all available information into the context window is problematic due to context window limitations, quadratic attention complexity, and the risk of 'context degradation' where models perform worse with longer inputs. Effective context engineering involves strategies like using a focused set of tools (avoiding prompt inflation), employing retrieval mechanisms to fetch only the most relevant information, and incrementally adding context over multiple turns rather than forcing an initial large insertion that invalidates the KV cache. The choice of how context is structured (e.g., Markdown, YAML) and the careful selection of tools also significantly impact agent performance.
Structured inputs/outputs and recursive prompting for agents
To facilitate interaction with agents and external tools, structured input/output formats are essential. Techniques like constrained decoding can enforce specific output structures (e.g., JSON), and frameworks like DSPy automate the creation of prompts that elicit structured responses. Furthermore, recursive prompting, where an LLM's output is fed back as input to itself or a sub-agent, offers a novel approach to handle long contexts and complex reasoning. By breaking down tasks into smaller, manageable sub-problems processed by child agents, this recursive structure can improve performance on tasks that traditionally suffer from context length limitations, such as the 'Ulong' benchmark.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Books
●Concepts
●People Referenced
Common Questions
The lecture breaks down large language models into three layers: 1. Model Engineering (focusing on distributed training and GPU kernels), 2. Model Adaptation (techniques like fine-tuning and distillation), and 3. Applications (building agents, chatbots, and RAG systems on top of LLMs).
Topics
Mentioned in this video
Neural networks used for language modeling, prominent around 2010.
Language models that existed before 2010, calculating probabilities based on the frequency of token sequences.
The neural network architecture that powers most modern large language models, introduced in the 'Attention is all you need' paper.
A memory cache used during the inference of language models to store key and value states, improving efficiency.
An optimization of the attention mechanism that reduces the quadratic complexity to linear, improving efficiency for long contexts.
A research paper that explored how language models can call tools, and also used for empirical evaluation of context formatting.
A language model that uses linear attention and shows degradation on longer sequences.
An open-source effort focused on training models or producing high-quality training data, involving extensive filtering of web data.
A framework for programming language models that allows defining inputs and outputs for structured generation.
A 32 billion parameter language model that showed significant improvement on the SweetBench benchmark after training on synthetic data.
A version of GPT models that shows improved results with increased inference time, as depicted in a performance curve.
A language model for coding tasks, tested in the 'Task Harnessing Tests' research.
A research paper and model that pioneered techniques for aligning language models with human preferences, leading to ChatGPT.
A language model known for its coding capabilities, tested in the 'Task Harnessing Tests' research.
A large language model that uses standard attention mechanisms.
A language model that utilizes linear attention and shows degradation on longer sequences.
A model mentioned in comparison to GPT-5 mini's performance on the Ulong test.
A model that combines linear attention with global attention layers to improve performance on tasks like retrieval.
A widely known chatbot developed by OpenAI, based on InstructGPT principles.
A dataset containing web crawl data used for training language models, from which a small percentage is filtered for quality.
A library that can generate finite state machines for decoding, helping to constrain LLM outputs.
A large, diverse dataset released publicly for training language models, including sources beyond the internet.
A benchmark used to evaluate the performance of language models, particularly in coding tasks.
A model developed by Thinking Machines that uses verifiable rewards for reinforcement learning.
A model that achieved top performance on the Ulong test by utilizing recurrent language modeling.
A benchmark for evaluating code generation capabilities of language models.
A benchmark for evaluating text summarization capabilities of language models.
A language model that shows a decline in accuracy on long-context tasks, as illustrated in a benchmark graph.
A model that uses a separate reward structure and trains multiple specialized models for different inference levels.
A benchmark for evaluating long-context performance of language models.
More from Stanford Online
View all 172 summaries
77 minStanford CS193T Thinking with AI | Autumn 2026 | Core Prompting
73 minStanford CS193T Thinking with AI | Autumn 2026 | Introduction to AI
70 minStanford CS329Z Introduction — What Are Agentic Systems?
65 minStanford AA274A Principles of Robotic Autonomy | Autumn 2019 | Extra Session - Particle Filter SLAM
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free