Key Moments
Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan
Key Moments
AI agents can be optimized through either prompts or weights, but prompt optimization is more sample-efficient and faster for frequent updates, especially with API-only models.
Key Insights
Context optimization is effective for 'last mile adaptation' to make models perform optimally for specific, nuanced tasks.
MiPROv2, a context optimization method, uses Bayesian search with program traces and instruction proposals to find optimal prompts.
On-policy distillation, a weight optimization technique, trains smaller models by sampling from a teacher model's rollouts and is more sample-efficient than SFT.
Reinforcement learning methods like PPO and GRPO reward agents for completing tasks, with GRPO being more memory-efficient by eliminating the need for a value model.
Genetic prompt evolution, as seen in JeePa, iteratively refines prompts by mutation or merging, aiming for Pareto optimal solutions across multiple tasks.
The "take a deep breath" prompt for PaLM 2L instruction-tuned models yielded better results than more complex prompts in one study.
The necessity of optimization for AI systems
Optimizing AI systems is crucial, requiring three key components: a system (like a language model or agent), data for improvement (rollouts, demonstrations, human feedback), and a metric for evaluation (pass rate, accuracy, cost, latency). Michael Ryan emphasizes that optimization allows for objective specification through data and evaluation, contrasting with traditional prompt engineering where objectives are manually encoded. The lecture highlights that while models improve with scale, context optimization remains vital for 'last mile adaptation'—fine-tuning a model for a specific, nuanced task where explicit instructions in prompts are insufficient. This adaptation can be achieved through careful system design or by setting up an evaluation framework and optimizing the system to perform well within it. Context optimization, as opposed to just prompt tweaking, is seen as a lasting technique for achieving this.
Context optimization techniques: Few-shot, bootstrapping, and reflective methods
Few-shot optimization leverages examples within prompts to guide models without weight updates. Early work like 'Fantastically Ordered Prompts' explored prompt ordering sensitivity, using a 'probing set' to measure how model outputs change based on example order. Modern approaches focus on bootstrapping reasoning, even when labels lack explicit chains of thought. Techniques like STaR (Search Tree Agent Reasoning) generate reasoning traces through rejection sampling, refining models through fine-tuning. DSPy's Bootstrap Random Search automates this by iteratively collecting high-scoring examples and combining them. Reflective optimization, exemplified by OPRO (Optimization by Reflection) and MiPROv2, involves using a meta-prompt to ask the model to improve its own prompts based on past performance. OPRO/MiPROv2 uses a Bayesian search strategy, incorporating program traces, instruction proposals, and context-aware grounding to iteratively refine prompts, demonstrating significant improvements over random search and manual prompting.
Advanced prompt optimization: TextGrad, GRPA, and the power of genetic algorithms
TextGrad is a library that defines programs as chains of calls and tools, using feedback from a language model to improve outputs via backpropagation. It frames optimization in deep learning terms, where the 'loss' is the score from an evaluation. Genetic Prompt Evolution (JeePa) uses evolutionary strategies like mutation and merging on prompts. It maintains a pool of candidate prompts, iteratively generating new ones based on performance (Pareto frontier) across multiple tasks. Prompts are refined through 'reflective prompt mutation' or 'system-aware merging,' aiming to improve performance on specific tasks or combine successful prompt components. This approach has shown state-of-the-art results across various benchmarks. The lecture also touches on the idea that agents themselves can be optimized through self-improvement, modifying their own code and iteratively refining their performance based on feedback and historical data, as seen in papers like 'Darwin's Godal Machine' and 'Auto-Research.'
Weight optimization: Supervised Fine-Tuning (SFT)
Supervised Fine-Tuning (SFT) is a foundational weight optimization technique. It involves training a model on specific data to match desired outputs. The core mechanism is cross-entropy loss, where the model is penalized for not predicting the correct next token. For agentic tasks, SFT can be applied by using a 'teacher model' to generate successful trajectories (demonstrations) for an agent. The student model is then trained on these successful rollouts. This pipeline involves collecting tasks from various sources, filtering them, selecting a teacher model (not necessarily the most capable, but one with similar capability to the student), generating rollouts, and finally fine-tuning the student model on these demonstrations. While SFT is effective with large datasets, it doesn't always account for the nuances of reinforcement learning where an agent learns through interaction and exploration.
Weight optimization: Reinforcement Learning (RL) and its variants
Reinforcement Learning (RL) trains agents by rewarding successful task completion. Key methods discussed are Proximal Policy Optimization (PPO) and Generalized Advantage Estimation Proximal Policy Optimization (GRPO). PPO uses four models in memory: policy, reward, reference, and value models, calculating rewards by combining KL divergence from a reference model and environmental rewards, then using Generalized Advantage Estimation (GAE) to update the policy. However, PPO is computationally expensive and memory-intensive. GRPO optimizes by eliminating the value model and using group advantages, reducing memory requirements to three models (policy, reward, reference) and simplifying the update process. This makes GRPO more efficient and practical. The reward signal can come from verifiable metrics (e.g., passing unit tests) or an LM judge if verification is difficult.
Weight optimization: Distillation and advanced RL techniques
Distillation techniques aim to transfer knowledge from a larger, capable 'teacher' model to a smaller, more efficient 'student' model. SFT can be seen as a form of hard distillation. On-policy distillation, however, leverages the teacher's conditional log probabilities for each token to guide the student, providing a richer learning signal than just the final output. This is particularly effective for training smaller models to mimic complex behaviors. The lecture also touches on policy distillation, where a model learns by observing expert demonstrations (like Grandmaster chess games) and potentially experimenting and receiving feedback, analogous to how humans learn. This is contrasted with simply observing expert actions, highlighting the benefit of active learning and exploration, especially when training frontier models for which no expert teacher exists.
Choosing the right optimization strategy
The choice between context and weight optimization depends on several factors. Context optimization is preferred when API access is the only option for powerful models, when frequent updates are needed (e.g., daily), or when interaction costs are high. It's faster for quicker iterations and often more sample-efficient. Weight optimization, particularly through SFT and RL, is suitable when a teacher model is available, when a cost-efficient model is desired, or when dealing with very large contexts that become unwieldy for prompt-based methods. The 'Better Together' paper suggests a hybrid approach, potentially alternating between prompt and weight optimization for optimal results. Ultimately, the goal is to leverage the strengths of each technique—sample efficiency and speed for context optimization, and deep adaptation for weight optimization—to build more capable and efficient AI agents.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Studies Cited
●Concepts
●People Referenced
Common Questions
To optimize an AI system, you need three key components: first, build your system; second, collect data for optimization; and third, establish a method to measure the impact of your optimization efforts.
Topics
Mentioned in this video
A framework or initiative that the speaker has closely worked with, particularly in the area of optimization.
Used to reverse-engineer Keynote's file structure and convert presentation data into the class's slide format, demonstrating its capabilities in code generation and file manipulation.
The presentation software used by the speaker, whose file structure was reverse-engineered by Codex.
Mentioned in the context of results from papers, with specific parameter sizes (2.7 billion and 175 billion).
Used in comparison to LLaMA 2 in the Fire Act paper, showing that fine-tuning with teacher-generated data can improve agent performance.
A series of models where on-policy distillation was used for smaller models, building upon larger teacher models.
A framework mentioned in relation to bootstrapped reasoning, and its paper from 2023 introduced bootstrap random search.
Used in the OPRO paper to optimize its own prompts, leading to better performance.
Mentioned as a previous version of GPT used to build an application, contrasting with future versions like GPT 6.1 or GPT 7.
Mentioned as a potential future version of GPT, implying ongoing model development.
Used in the OPRO paper for optimization, it generated a prompt that outperformed the original Chain of Thought prompt.
A method for genetic prompt evolution, involving mutation and merging of prompts to optimize performance.
Mentioned as a potential future version of GPT, implying ongoing model development.
A specific version of LLaMA 2 that showed improved agent performance when fine-tuned using the Fire Act method.
Mentioned as a potential teacher model for a 3B parameter student model, highlighting the importance of matching teacher capability to student capability.
A model that was fine-tuned using trajectories generated by GPT-4 in the Fire Act paper.
A model series developed using RL and SFT, demonstrating the creation of a reasoning-capable model from scratch.
A classic machine learning dataset used as an example for classification tasks in the MiPRO v2 paper.
A fundamental concept in supervised fine-tuning, used to measure the difference between predicted and actual token probabilities.
Refers to a model size of 3 billion parameters, used as an example when discussing teacher-student model capabilities in distillation.
A reasoning design pattern discussed in the Fire Act paper for fine-tuning agents.
A paper that provided an example of an agent task involving file searching and error capture, used to illustrate SFT.
A research paper that demonstrates how to generate reasoning paths through rejection sampling, allowing for bootstrapped reasoning without explicit human-provided logic.
A research paper on privacy-preserving inference, used as an example for how Jeeppa can be applied to generate privacy-preserving requests.
A research paper led by the speaker, which combines demonstration bootstrapping with Bayesian search for prompt optimization.
A paper that introduces the concept of an agent modifying its own harness code for self-improvement.
A paper that focused on fine-tuning agents, collecting questions and running different reasoning design patterns like Chain of Thought and ReAct.
A paper that introduced the GRPO method.
A research paper suggesting that weight and prompt optimization can work together to solve tasks.
More from Stanford Online
View all 184 summaries
59 minHow AI Agents Work: Connectors, MCP & Agent Safety | Stanford CS193T Lecture 5 with Cynthia Bailey
70 minVibe Coding, AI Risk & AI Evaluation | Stanford CS193T Lecture 6 with Cynthia Bailey
79 minMulti-Agent Systems: When More Agents Help or Hurt | Stanford CS329Z Lecture 8 with Michael Ryan
79 minHow AI Agent Memory Works: MemGPT & Memory Types | Stanford CS329Z Lecture 7 with Michael Ryan
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free