Key Moments
Stanford CS329A Self-Improving AI Agents | Part 4 | Learning from Feedback with Tools/Code
Stanford OnlineWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI agents can now learn from their mistakes and critiques using tool calling, execution feedback, and AI-generated feedback, dramatically improving their capabilities beyond simple reasoning or action.
Key Insights
The React framework combines reasoning and action in LLMs, allowing them to interact with tools and environments, improving performance on knowledge-intensive tasks like question answering and fact-checking.
RLEF (Reinforcement Learning from Execution Feedback) uses test execution results to fine-tune coding LLMs, achieving higher solve rates on competitive programming tasks than base models or supervised fine-tuning alone.
Constitutional AI enables LLMs to improve harmlessness and helpfulness by using AI-generated feedback based on human-written principles, reducing reliance on extensive human labeling.
React agents demonstrated a 66.6% success rate on the web shop task, significantly lower than human experts at 82.1%, highlighting the continued challenges in complex decision-making tasks.
In RLEF, using public tests for immediate feedback during training significantly improves code generation, while private tests are reserved for final reward calculation to prevent memorization.
Constitutional AI, particularly when combined with chain-of-thought reasoning, achieved the best Pareto frontier for balancing helpfulness and harmlessness compared to standard RLHF.
Integrating reasoning and action with the React framework
Large Language Models (LLMs) excel at natural language processing but require interaction with real-world tools and code to become truly useful. The React framework addresses this by combining reasoning and action, mimicking human thought processes. Humans first think about a task, reason about potential actions, execute those actions, observe the outcomes, and then update their reasoning. React applies this loop to LLMs. Unlike traditional LLMs that might hallucinate or lack real-world grounding, React enables models to generate verbal reasoning traces, take actions (like tool calls), and incorporate observations back into their reasoning. This is crucial for tasks requiring real-world knowledge, such as hotspot QA, fact-checking (Fever), and interactive tasks like online shopping (WebShop). A key benefit of this explicit reasoning-action loop is increased interpretability, allowing humans to trust the model's responses more readily. While current models often incorporate this implicitly, React provides a foundational abstraction for grounding LLM outputs in external knowledge and actions.
Grounding code generation with execution feedback (RLEF)
For coding agents, execution feedback from running code and tests is vital for iterative improvement. The RLEF (Reinforcement Learning from Execution Feedback) framework demonstrated that this feedback significantly enhances LLM performance in code generation. It operates through an end-to-end RL fine-tuning process where generated code is evaluated against test cases. When tests fail, the execution feedback (pass/fail status) is fed back to the LLM, allowing it to refine its code in subsequent turns. This iterative process continues until the code passes a set of tests or a turn limit is reached. A key innovation is the two-tier test strategy: public tests provide immediate, rapid feedback during inference for faster iteration, while private tests, kept hidden, are used for the final reward calculation in the PPO training loop. This separation prevents the model from simply memorizing test outputs and ensures more robust generalization. RLEF has shown improved solve rates on competitive programming tasks, demonstrating the power of learning from errors in a structured feedback loop.
Improving harmlessness and helpfulness with Constitutional AI
Collecting human feedback for training LLMs, especially for fine-tuning harmlessness and helpfulness, is time-consuming and expensive. Constitutional AI offers a scalable alternative by using AI-generated feedback based on human-written principles, termed a 'constitution.' Humans define a set of principles (e.g., avoid harmful or unethical content, be respectful) and the AI uses these principles to critique and revise its own outputs. The process involves two stages: supervised fine-tuning, where the model learns from self-critique and revision examples, and reinforcement learning, where an AI-generated preference model trained on the constitution fine-tunes the LLM. This approach allows models to improve their harmlessness significantly, often at the cost of a slight decrease in helpfulness, though the combined helpfulness and harmlessness can improve monotonically. Constitutional AI, especially when combined with chain-of-thought reasoning, has shown promising results in achieving a better trade-off between these two crucial aspects compared to standard RLHF.
React's performance and limitations
The React framework has shown notable success in grounding LLMs and improving their performance on knowledge-intensive tasks like question answering and fact-checking, and decision-making tasks such as navigating a simulated web shop. For instance, on the web shop task, React agents achieved a 66.6% success rate, which is an improvement over imitation learning but still trails human expert performance at 82.1%. This highlights that while React provides a valuable structure for combining reasoning and action, challenges remain, particularly in complex, multi-step decision-making processes where errors can cascade. A significant limitation mentioned is that for tasks with very large action spaces, React requires extensive demonstrations, which can be challenging to fit within context windows, and its multi-step reasoning increases inference costs.
RLEF's effectiveness in code generation
RLEF's application to code generation, particularly for competitive programming tasks, demonstrated substantial gains. Even with simpler models like LLaMA 3.1, RLEF significantly increased solve rates compared to baseline methods. The framework leverages the errors made by the model to guide its learning; by analyzing the specific mistakes and edits required, the model learns to produce more targeted and correct code. This iterative refinement process, informed by execution feedback, not only reduces the sampling budget needed to achieve state-of-the-art performance but also helps the model generalize to unseen benchmarks. The use of binary rewards from test failures has proven sufficient for these tasks, although the potential need for more detailed error traces for complex problems is acknowledged as an area for future research.
Constitutional AI: Scaling feedback loops
Constitutional AI addresses the scalability issue of human feedback by creating an AI-driven loop. By defining a 'constitution' of principles, models can self-critique and revise their outputs, reducing the need for extensive human labeling. While supervised fine-tuning on self-critique data improves harmlessness, it can potentially decrease helpfulness if not balanced. The RL stage, using an AI-trained preference model based on the constitution, aims to optimize both helpfulness and harmlessness. This method has been shown to push the Pareto frontier, offering a more balanced approach than solely optimizing for helpfulness or harmlessness independently. The challenge of continually updating or amending the constitution and ensuring models 'forget' old rules remains an active research area, with implications for continual learning.
Future directions and challenges in self-improving AI
The lecture highlights several ongoing research directions. The integration of cognitive science with LLMs is crucial for developing more sophisticated AI reasoning and problem-solving mechanisms. For agents, understanding the interplay between LLM capabilities and agentic features like memory and sessions is key. Handcrafted frameworks like React might become less dominant as RL-based methods advance, but their underlying workflow definitions remain valuable, especially for domain-specific tasks where defining general search spaces is difficult. Challenges such as noisy environments, the need for backtracking, and developing better confidence metrics for AI feedback are active areas. Furthermore, the debate between process-reward and outcome-reward models, as well as the nuances of generalization versus memorization in coding agents, continue to drive innovation in self-improving AI.
Mentioned in This Episode
●Products
●Software & Apps
●Concepts
Common Questions
React combines reasoning (thinking) and acting (tool calls) in a loop. This allows LLMs to interact with the environment, get observations, and use that feedback to refine their reasoning and take better actions, leading to more grounded and interpretable outputs.
Topics
Mentioned in this video
A framework that combines reasoning and acting in language models, allowing them to take into account actions via tool calls.
A technique where a model uses its own critique as a feedback loop to improve itself, particularly for harmlessness.
A reasoning technique for LLMs where the model shows intermediate steps to reach an answer, but these steps may not be grounded without external feedback.
A fact-checking task that requires real-world knowledge and grounding, used to evaluate models like React.
A task where an agent is supposed to buy products on the internet, requiring a series of tool calls and actions.
A language model mentioned as an example of a model that has no information about current events unless it performs a search call.
A language model mentioned as an example of a model that has no information about current events unless it performs a search call.
A language model mentioned as an example of a model that has no information about current events unless it performs a search call.
An older model used to implement the React idea with a frozen LLM, demonstrating few-shot examples of actions, thoughts, and observations.
A reinforcement learning algorithm mentioned as being usable for the RLEF approach, particularly for updating the policy based on execution results.
An older model on which RLEF results were demonstrated, showing improved solve rates with execution feedback.
A benchmark that targets problems related to searching for relevant information and applying code edits, relevant to cloud code functionalities.
More from Stanford Online
View all 116 summaries
75 minStanford CS329A Self-Improving AI Agents | Part 5 | Planning and Multi-Step Reasoning
73 minStanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL
76 minStanford CS329A Self-Improving AI Agents | Part 8 | Agentic Evaluations and Long Horizon Tasks
73 minStanford CS329A Self-Improving AI Agents | Part 3 | Robust Verification
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free