How is the policy gradient derived?

The policy gradient is derived by taking the gradient of the RL objective with respect to the policy parameters. This involves using sampling to approximate expectations and leveraging logarithmic derivative tricks to obtain a computable expression.

What is the REINFORCE algorithm?

REINFORCE is one of the first policy optimization algorithms. It uses the derived policy gradient to iteratively update the policy parameters by sampling trajectories, estimating returns, and taking gradient ascent steps.

Why is policy gradient estimation high in variance?

Policy gradient estimation suffers from high variance because it relies on sampled trajectories and their associated noisy returns. The performance of these samples can vary significantly, leading to an unstable learning signal.

How can the variance of policy gradients be reduced?

Variance can be reduced by incorporating causality into the reward-to-go calculation (summing rewards only from the current time step onwards) and by introducing baselines, which help to center the returns and provide a more stable learning signal.

What is a baseline in policy gradient methods?

A baseline is a quantity subtracted from the trajectory return to reduce the variance of the policy gradient estimator. Crucially, adding a baseline does not change the expected value of the gradient, making it an unbiased method for variance reduction.

What are actor-critic methods?

Actor-critic methods combine policy gradient (the actor) with value-based methods (the critic). The critic estimates value functions to provide a more accurate estimate of the reward-to-go, thereby reducing the variance in the actor's policy updates.

What is the advantage function?

The advantage function quantifies how much better an action is compared to the average action in a given state. It is often used in actor-critic methods as a baseline to improve policy updates.

How does Advantage Actor-Critic (A2C) work?

A2C fits a value function (the critic) and uses it to compute the advantage, which is then used in the policy gradient updates for the actor. It simplifies the estimation by only requiring a value function fit.

What was AlphaGo's contribution to reinforcement learning?

AlphaGo demonstrated a successful application of actor-critic methods, policy gradients with baselines, and Monte Carlo Tree Search to achieve super-human performance in the complex game of Go.

Are policy gradient methods on-policy or off-policy?

In their basic form, policy gradient methods like REINFORCE are on-policy, meaning they require data collected from the current policy to estimate the gradient. Extensions exist to make them more off-policy.

Key Moments

Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 18: RL Policy Optimization

Stanford Online

Education5 min read81 min video

Aug 13, 2026|397 views|16

Stanford Stanford Online Robotics

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Policy optimization directly tunes policy parameters to maximize rewards, unlike value-based methods. However, its gradient estimators suffer from high variance, necessitating techniques like baselines and actor-critic methods for stability and efficiency.

Key Insights

Policy optimization directly optimizes a parametric policy's parameters (theta) to maximize the reinforcement learning objective, contrasting with value-based methods that implicitly define the policy.

The policy gradient estimator is derived as the expectation of the gradient of the log probability of actions multiplied by the sum of rewards, which can be approximated through sampling rollouts.

The REINFORCE algorithm, a foundational policy optimization method, involves sampling trajectories, estimating the policy gradient, and taking a gradient ascent step in parameter space.

High variance in policy gradient estimators is a major challenge, leading to unstable training; techniques like using the reward-to-go from the current time step onwards (instead of the entire trajectory) and introducing baselines can reduce this variance.

Actor-critic methods combine policy optimization (actor) with value function estimation (critic) to provide a lower-variance estimate of the policy gradient, often using the advantage function (Q-function minus V-function) as a baseline.

The AlphaGo paper utilized an actor-critic approach with a policy network trained via policy gradient (REINFORCE with a value function baseline) and a value network, leveraging self-play and Monte Carlo tree search.

Shifting from value-based to policy optimization

This lecture introduces policy optimization, a family of model-free reinforcement learning methods that directly optimize a policy's parameters. Unlike value-based methods where the policy is implicitly derived from value functions, policy optimization represents the policy parametrically (pi(theta)) and tunes these parameters (theta) to directly maximize the reinforcement learning objective. This approach frames reinforcement learning as an optimization problem, utilizing gradient ascent to iteratively adjust policy parameters in the direction that increases expected cumulative rewards.

Deriving the policy gradient

The core of policy optimization lies in calculating the gradient of the RL objective with respect to policy parameters. The objective J(theta) is the expected sum of discounted rewards over all possible trajectories, denoted by P(tau). To make this computable, the gradient is manipulated using the identity P(tau) * grad(log P(tau)) = grad(P(tau)). This leads to an expression involving the expectation of the gradient of the log-policy multiplied by the sum of rewards. This expectation, which is otherwise intractable due to unknown environment dynamics, can be approximated by sampling trajectories (rollouts) from the environment using the current policy and computing the empirical mean.

The REINFORCE algorithm

The REINFORCE algorithm is a direct implementation of this policy gradient approach. It involves three main steps: 1. Sampling multiple trajectories (rollouts) by interacting with the environment using the current policy pi(theta). 2. Estimating the policy gradient using these sampled trajectories, where the gradient is approximated by the empirical average of (grad log pi(theta) * sum of rewards). 3. Updating the policy parameters theta using gradient ascent with this estimated gradient. This process is typically repeated iteratively, with the policy being updated and new experience collected based on the improved policy.

Addressing high variance in policy gradients

A significant challenge with the basic policy gradient estimator is its high variance. This means the estimated gradient can fluctuate wildly depending on the sampled trajectories, leading to unstable and slow learning. The lecture highlights that this variance stems from the noisy estimates of the cumulative rewards (reward-to-go). Two strategies are presented to mitigate this: 1. Causality: Using only rewards from the current time step onwards ('reward-to-go') in the gradient calculation, as actions cannot influence past rewards. 2. Baselines: Subtracting a baseline value (B) from the reward-to-go. The crucial insight is that subtracting a baseline does not change the expected value of the policy gradient estimator (it remains unbiased), but it can significantly reduce variance. A common choice for the baseline is the average return observed across sampled trajectories.

Actor-Critic methods

Actor-critic methods aim to combine the strengths of policy optimization (the 'actor') and value-based methods (the 'critic'). The critic estimates the value function (V(s) or Q(s,a)), providing a more stable and lower-variance estimate of the 'reward-to-go' than simply using sampled returns. Instead of multiplying the log-policy gradient by the sampled return, actor-critic methods use the critic's estimate. This often takes the form of the advantage function A(s,a) = Q(s,a) - V(s), which quantifies how much better an action is compared to the average action in that state. This advantage function serves as a learned, state-dependent baseline, further reducing variance and improving learning efficiency.

Actor-critic implementation and AlphaGo

The implementation of the critic (value function estimation) in actor-critic methods mirrors techniques used in value-based learning, such as fitting a parametric function approximator using Monte Carlo or Temporal Difference (TD) targets. A popular variant is the Advantage Actor-Critic (A2C) algorithm, which approximates the advantage function by fitting only the state-value function V(s), simplifying the estimation process. The lecture also references the AlphaGo system as a prime example of actor-critic methods in practice, using a policy network (actor) trained via policy gradients with a value function baseline and a value network (critic) for game state evaluation.

Pros and cons of policy gradient methods

Policy gradient methods naturally handle continuous action spaces, unlike traditional value-based methods that often rely on discrete actions for argmax operations. They directly optimize the desired RL objective, providing a clearer measure of progress during training. However, the basic algorithms discussed are typically on-policy, meaning they require new samples for each policy update, which can lead to sample inefficiency. The high variance of the policy gradient estimator is also a critical drawback, although mitigated by techniques like baselines and actor-critic architectures.

Mentioned in This Episode

●Software & Apps

●Concepts

Common Questions

Policy optimization directly learns a parametric policy by adjusting its parameters to maximize the reinforcement learning objective. Unlike value-based methods where the policy is implicit, policy optimization represents and tunes the policy explicitly.

Topics

Reinforcement Learning AI & Machine Learning Technology & Innovation Science & Mathematics Policy Optimization Policy Gradients Value Functions Actor-critic Methods Variance Reduction

Mentioned in this video

Concepts

Monte Carlo

A fundamental model-free learning strategy for estimating value functions in reinforcement learning.

Temporal Difference Learning

A fundamental model-free learning strategy for estimating value functions in reinforcement learning.

Markov decision process

A formalism used in reinforcement learning to define the problem, involving states, actions, and transitions.

Monte Carlo Tree Search

A search algorithm used in conjunction with policy optimization in AlphaGo to guide action selection.

Software & Apps

Gym Environments

A common toolkit for developing and comparing reinforcement learning algorithms, used as an example for initial state sampling.

Reinforce algorithm

The first policy optimization algorithm discussed, serving as a foundational example for policy gradients.

Q-learning

A value-based reinforcement learning algorithm contrasted with policy optimization methods.

Media

AlphaGo

A project that demonstrated the success of reinforcement learning, particularly actor-critic methods, in playing the game of Go.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free