What are TRPO and PPO and why were they developed?

TRPO (Trust Region Policy Optimization) and PPO (Proximal Policy Optimization) are algorithms that aim to improve policy optimization by ensuring stable updates. They were developed to address limitations in earlier policy gradient methods and simplify optimization problems.

How does PPO work with its clipped surrogate objective?

PPO uses a clipped surrogate objective function that penalizes large policy changes. It minimizes the difference between the ratio of new and old policy probabilities multiplied by the advantage, or a clipped version of this term, to maintain stability.

What are the main paradigms of model-free reinforcement learning discussed?

The main paradigms are Monte Carlo learning, which uses full sampled returns, and Temporal Difference (TD) learning, which uses bootstrapping for lower variance and online learning. Both can be applied to tabular representations or function approximation.

What is the basic recipe for model-based reinforcement learning?

The basic recipe involves interacting with the environment to collect data, using this data to fit a model of the environment's dynamics, and then using this learned model for planning or control.

What are the key challenges in model-based RL?

A major challenge is extrapolation from observed data to unseen states, leading to distribution shift and potential exploitation of errors in the learned model, especially with complex dynamics and high-capacity models like neural networks.

Why is uncertainty quantification important in model-based RL?

Uncertainty quantification helps mitigate the risk of exploiting errors in learned models. By estimating how certain the model is about its predictions, planning algorithms can avoid actions that might lead to poor outcomes in uncertain regions of the state space.

What is the difference between aleatoric and epistemic uncertainty?

Aleatoric uncertainty is inherent noise in the process being observed, while epistemic uncertainty arises from uncertainty about the model itself, such as its parameters or structure.

How can Gaussian Processes model uncertainty?

Gaussian Processes model uncertainty by defining a distribution over functions. They provide an analytical expression for the exact posterior distribution over functions consistent with the observed data, allowing for data-efficient estimation.

How do bootstrap ensembles estimate uncertainty?

Bootstrap ensembles train multiple independent models on the same data. Disagreement among these models, especially in regions far from the training data, indicates uncertainty.

What are the main benefits of model-based RL compared to model-free methods?

Model-based RL, particularly when incorporating uncertainty quantification, often demonstrates significantly higher sample efficiency, meaning it can achieve good performance with fewer interactions with the environment.

How are different control algorithms combined in practice, for example, in autonomous driving?

Control algorithms are typically combined hierarchically, with high-level decision-making (e.g., goal setting) using methods like dynamic programming, trajectory generation using open-loop techniques, and fine-grained control and tracking using model predictive control or PID controllers.

Key Moments

Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 19: Model-Based RL

Stanford Online

Education4 min read82 min video

Aug 13, 2026|519 views|10

Stanford Stanford Online Robotics

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Model-based RL, while promising for sample efficiency, struggles with complex dynamics and model uncertainty. New methods use uncertainty quantification and ensembles to mitigate these issues, but model-free approaches like PPO remain more mature for continuous control.

Key Insights

Model-based RL requires learning a dynamics model, which can be challenging for complex, non-linear systems and high-capacity models like neural networks, leading to extrapolation errors.

Trust Region Policy Optimization (TRPO) introduced a constrained optimization problem using KL divergence to keep policy updates within a 'trust region', but its implementation with conjugate gradient methods was complex.

Proximal Policy Optimization (PPO) simplifies TRPO by using a clipped surrogate objective function to empirically approximate the behavior of the constraint, making it more practical and popular.

Uncertainty quantification in model-based RL aims to estimate how certain a learned model is about its predictions, allowing for better planning by reasoning about expected rewards under model uncertainty.

Gaussian Processes offer data-efficient uncertainty estimation by providing a distribution over functions, but their scalability to high-dimensional data is a challenge.

Bootstrap ensembles train multiple models independently to capture uncertainty, with agreement indicating certainty and disagreement signaling uncertainty, providing an empirical approximation of the posterior distribution over model parameters.

The challenge of learning dynamics models

Model-based reinforcement learning (MBRL) fundamentally relies on learning a model of the environment's dynamics, which predicts the next state given the current state and action. This learned model can then be used with optimal control techniques. However, fitting accurate models for complex, non-linear dynamics, especially with high-capacity models like neural networks, is difficult. A key issue is extrapolation – when the learned model is used for planning, it may encounter states outside the distribution of the training data, leading to poor generalization and potentially exploiting errors in the model's predictions in favorable directions.

From TRPO to PPO: refining policy optimization

The lecture first revisits model-free RL, introducing Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). TRPO aimed to solve reinforcement learning as a numerical optimization problem by defining a surrogate objective and adding a constraint on the KL divergence between old and new policies to ensure updates stayed within a 'trust region'. While successful, TRPO's reliance on second-order optimization methods like conjugate gradient made it complex to implement and less effective with deep neural networks. PPO emerged as a simplification, aiming to achieve similar stability without the complex optimization. It introduces a clipped surrogate objective function that constrains the policy update ratio, effectively limiting how much the policy can change in a single step and providing a more practical and widely adopted approach for policy optimization in reinforcement learning.

The role of uncertainty quantification in model-based RL

A central theme in MBRL is dealing with the uncertainty inherent in learned models. When a model is used for planning, uncertainty quantification helps in understanding the model's confidence in its predictions. Instead of a single point estimate for the next state, a model that outputs a distribution allows planners to reason about expected outcomes and potential risks. For example, when planning trajectories near a cliff, a model with high uncertainty about its terminal state might correctly identify a safer path than one that is confidently, but incorrectly, predicting a safe landing.

Modeling uncertainty: Gaussian processes and ensembles

Two primary approaches for modeling uncertainty are Gaussian Processes (GPs) and bootstrap ensembles. GPs provide a distribution over functions, offering data efficiency and analytical posterior expressions, making them strong for low-data regimes. However, they can be computationally intensive for high-dimensional data. Ensembles, on the other hand, train multiple models independently. Disagreement among ensemble members signals uncertainty, while agreement suggests confidence. By averaging predictions from these models, ensembles offer an empirical way to approximate the uncertainty in predictions, particularly effective at capturing multiple modes in the posterior distribution of model parameters.

Integrating learned models with planning

To leverage learned models with uncertainty estimates, strategies like sampling model parameters from the posterior distribution are used. For a given action sequence, the model can be propagated multiple times using different sampled parameter sets, generating various predicted futures and their associated rewards. Averaging these rewards provides an estimate of the expected reward under model uncertainty. This approach is exemplified by algorithms like PETS (Probabilistic Ensembles with Trajectory Sampling), which uses an ensemble of neural networks for the dynamics model and a model predictive control (MPC) framework with a cross-entropy method to plan actions in a receding-horizon fashion. This often leads to greater sample efficiency compared to model-free methods.

Model-based vs. model-free: a pragmatic view

While model-based RL shows promise, particularly in sample efficiency, model-free methods like PPO and Soft Actor-Critic are currently more mature and widely adopted, especially for continuous control tasks. The choice often depends on the specific application and the complexity of the dynamics. Model-free algorithms are generally more robust and easier to implement, whereas model-based methods require careful handling of model learning and uncertainty.

Hierarchical decision-making in autonomy

Autonomous systems often employ a hierarchical decision-making structure. This typically starts with a high-level goal (e.g., changing lanes), often handled by dynamic programming-like methods to account for stochasticity. This goal is then translated into a fine-grained trajectory using open-loop planning, which may use lower-fidelity models. Model Predictive Control (MPC) is used for tracking this trajectory, incorporating more detailed system dynamics and safety constraints. Finally, low-level controllers (like PID) actuate the system. Learning-based and end-to-end approaches are increasingly being integrated into this hierarchy, often carving out space between open-loop planning and trajectory tracking.

Mentioned in This Episode

●Software & Apps

●Concepts

Common Questions

Model-free RL learns policies or value functions directly from experience without explicitly modeling the environment's dynamics. Model-based RL first learns a model of the environment's dynamics and then uses this model for planning or control.

Topics

Reinforcement Learning AI & Machine Learning Science & Mathematics Policy Optimization Robotics Control Model-based RL Optimal Control Model-free RL Uncertainty Quantification

Mentioned in this video

Software & Apps

REINFORCE

An algorithm for policy optimization in model-free reinforcement learning that uses policy gradients.

TRPO

Trust Region Policy Optimization, an algorithm designed to improve policy optimization by ensuring updates stay within a trust region, preventing large, destabilizing changes.

PPO

Proximal Policy Optimization, a derivative of TRPO that simplifies implementation while maintaining policy stability and performance, widely used in reinforcement learning.

Sarsa

An on-policy temporal difference learning algorithm for reinforcement learning, used as an example of value-based methods.

Q-learning

A model-free reinforcement learning algorithm that uses value-based methods to learn optimal actions by estimating the quality of state-action pairs.

Deep Q-Networks

An extension of Q-learning that uses deep neural networks to approximate the Q-value function, enabling it to handle high-dimensional state spaces.

Gaussian Processes

A non-parametric model-based approach used for estimating uncertainty in predictions by defining a distribution over functions, particularly useful in low-data regimes.

PETS

A model-based reinforcement learning algorithm that uses an ensemble of neural networks to approximate the posterior over models and incorporates uncertainty quantification for planning.

Soft Actor-Critic

An off-policy actor-critic reinforcement learning algorithm that uses entropy maximization to encourage exploration and improve robustness.

Concepts

Actor-Critic

A class of reinforcement learning algorithms that combine policy-based and value-based methods, often used with policy optimization.

KL Divergence

A measure of how one probability distribution diverges from a second, expected probability distribution. Used in TRPO to constrain policy updates.

Monte Carlo learning

A model-free reinforcement learning paradigm that uses sampling from the environment to approximate expectations and update value functions based on observed returns.

Temporal Difference (TD) learning

A model-free reinforcement learning method that combines sampling from Monte Carlo with bootstrapping from dynamic programming, updating value estimates based on one-step predictions.

Bootstrap ensembles

A method for estimating uncertainty by training multiple models independently on the same data and observing their agreement or disagreement, which indicates confidence levels.

Model Predictive Control

A control strategy that uses a model of the system to predict future states and optimize control actions over a finite horizon, re-planning at each step.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free