How does the gradient of log probability simplify when conditioning on past states?

When conditioning on past states and actions, the gradient of the log probability of the current action becomes zero due to the Markov property and the fact that the gradient of a log probability distribution with respect to its parameters, averaged over samples from that distribution, is zero.

What is the purpose of introducing a baseline in policy gradient methods?

A baseline is introduced to reduce the variance of the policy gradient estimator. While it doesn't change the expected value of the gradient, it can significantly decrease the empirical variance across different trajectories, leading to more stable and efficient learning.

Why is standard policy gradient considered 'on-policy'?

Standard policy gradient methods are 'on-policy' because they require samples to be generated from the current policy being updated. This means collected data cannot be directly reused for subsequent updates without resampling, limiting sample efficiency.

How does PPO address the limitations of on-policy learning?

PPO uses importance sampling to allow the reuse of samples collected from older policies. It corrects for the distribution shift between the old and new policies using a ratio term, combined with a clipped surrogate objective function to prevent excessively large updates.

How can reinforcement learning be used to train LLMs for chain of thought?

Reinforcement learning can train LLMs to generate better chain of thought by treating the generation process as an MDP. Rewards are given based on the correctness of the final answer, allowing the model to learn optimal reasoning paths without explicit supervision on intermediate steps.

What is the role of the reward function in RL for LLM reasoning?

The reward function in RL for LLM reasoning typically evaluates the final answer against a ground truth. It's often binary (1 for correct, 0 for incorrect) but can incorporate factors like answer format or employ another LLM for evaluation, though this risks being 'hackable'.

Why is it crucial to achieve non-zero rewards early in RL training for LLMs?

Starting with a policy that yields at least some non-zero rewards is critical. If the initial policy always produces zero rewards, the reinforcement learning process struggles to identify beneficial behaviors and bootstrap the learning of desired reasoning capabilities.

What are the computational trade-offs between generation and training in RL for LLMs?

Generation of trajectories in RL for LLMs is typically slower than training because it's token-by-token and harder to parallelize. Training, on the other hand, involves feeding pre-generated trajectories through the model, which is more amenable to parallel computation.

Key Moments

Stanford CS229 Machine Learning | Spring 2026 | Lecture 20: GMM (EM), PCA

Q: What is the key idea behind the 'chain of thought' prompting technique?

Chain of thought prompting encourages large language models to generate intermediate reasoning steps or 'thinking tokens' before providing a final answer. This technique, discovered through few-shot examples and later refined to zero-shot prompting, significantly improves performance on tasks requiring multi-step reasoning.

Stanford Online

Education5 min read79 min video

Jul 31, 2026|714 views|18

Stanford Stanford Online Machine Learning Artificial Intelligence AI

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Reinforcement learning for LLMs is notoriously unstable, leading to algorithms like PPO and Siso that use clipping to prevent drastic policy updates, but their theoretical justifications remain debated.

Key Insights

The policy gradient theorem allows for optimizing policies by weighting log probabilities of actions by rewards, but requires careful estimation to avoid zero gradients.

Subtracting a baseline from the reward-to-go can significantly reduce variance in policy gradient estimation without altering the expected gradient.

Proximal Policy Optimization (PPO) uses a clipped surrogate objective to limit the magnitude of policy updates, preventing drastic changes and improving stability.

The Siso algorithm further modifies the PPO objective by clipping both positive and negative advantages, aiming to balance learning stability with policy improvement.

Chain-of-thought prompting significantly improves LLM performance on complex reasoning tasks by encouraging step-by-step thinking, a capability that can be further enhanced through reinforcement learning.

Reinforcement learning for LLMs often uses the final answer as the reward signal, treating the generation process as an MDP where actions are token predictions, though designing effective reward functions and baselines remains a challenge.

Understanding the policy gradient theorem

The lecture begins by revisiting the policy gradient algorithm, which aims to optimize a policy by adjusting its parameters (theta) to maximize expected rewards. The core challenge is that the reward function is not directly dependent on theta; instead, theta influences the probability of sampling trajectories. The policy gradient estimator uses the gradient of the log probability of sampled actions, weighted by the total reward. This weighting mechanism intuitively means that trajectories with higher rewards contribute more to the gradient, guiding the policy towards better actions. A crucial mathematical property is that the expectation of the gradient of the log probability of an action, when averaged over actions sampled from the same distribution, is zero. This is because, without any weighting, there's no inherent preference for any action. This property is foundational for deriving and simplifying policy gradient updates.

Reducing variance with baselines

A significant technique discussed to stabilize policy gradient methods is the introduction of a baseline. The lecture explains that adding or subtracting a term that is independent of the sampled action (e.g., a function of the current state) to the reward does not change the expected gradient. However, it can drastically reduce the variance of the gradient estimate. This is achieved by subtracting a baseline, such as the average reward-to-go, from the actual reward. This adjustment shifts the reward values but maintains their relative order, making the gradient estimates more stable. For example, if rewards are always between 10 and 11, subtracting 10.5 centers them around zero, reducing the impact of extreme values and leading to more consistent updates. This baseline subtraction is a critical component for practical reinforcement learning, especially in complex environments.

Proximal Policy Optimization (PPO) for stability

The lecture then delves into Proximal Policy Optimization (PPO), an algorithm designed to address the instability issues often encountered with policy gradients. PPO aims to prevent excessively large policy updates by introducing a clipped surrogate objective function. This clipping mechanism limits how much the new policy can deviate from the old policy during an update. Specifically, PPO considers the ratio between the probability of an action under the new policy and the old policy. If this ratio, combined with the advantage function (reward minus baseline), suggests a large positive update (i.e., a good action with a significantly increased probability), PPO clips the objective to prevent overly aggressive learning. Similarly, for negative advantages (bad actions), it also applies clipping. This careful control over update magnitude is key to PPO's effectiveness and popularity in training large models.

The Siso algorithm and its modifications

Following PPO, the lecture introduces Siso, another popular algorithm that builds upon PPO's principles. Siso modifies the clipping strategy. On the positive advantage side (good actions), if the advantage is high and the probability ratio exceeds a certain threshold (e.g., 1 + epsilon_high), Siso still caps the update, but it does so by setting the objective to a constant value rather than zeroing it out entirely, allowing for some continued learning. On the negative advantage side (bad actions), Siso's approach differs more significantly. While PPO might zero out updates if the action is bad and its probability is already low, Siso introduces a different clipping mechanism that aims to prevent drastic penalties, potentially preserving policy diversity. The rationale behind these modifications is complex and still an active area of research, with empirical results often guiding their adoption.

Chain-of-Thought prompting and its benefits

The latter part of the lecture shifts to applying these concepts to Large Language Models (LLMs), focusing on 'chain-of-thought' (CoT) prompting. CoT involves prompting the LLM to generate intermediate reasoning steps before providing a final answer. This technique, discovered through few-shot and later zero-shot prompting (e.g., by adding phrases like 'Let's think step by step'), dramatically improves performance on complex tasks requiring multi-step reasoning, such as math problems. It allows the model to break down problems, show its work, and arrive at more accurate conclusions. The effectiveness of CoT highlights the importance of the LLM's internal thought process, not just its final output.

Reinforcement learning for training LLM reasoning

To further enhance LLMs' reasoning capabilities, reinforcement learning (RL) is employed. Instead of relying solely on supervised fine-tuning, which requires extensive hand-crafted step-by-step thinking examples, RL allows models to learn from rewards based on the final answer's correctness. The LLM generation process is framed as a Markov Decision Process (MDP), where states are the current sequence of tokens and actions are the generation of the next token. A reward is assigned only at the end of the sequence, typically based on whether the final answer matches a ground truth. Algorithms like PPO and Siso are then used to train the LLM policy to generate sequences that maximize this final reward. A critical challenge is ensuring the initial policy can generate at least some trajectories with non-zero rewards, often requiring pre-training or careful reward shaping.

Mentioned in This Episode

●Software & Apps

Common Questions

The core challenge is that the gradient calculation requires swapping the gradient operator with an expectation, which is not permissible when the policy parameters are part of the probability distribution being sampled from, leading to a zero gradient if not handled carefully.

Topics

Reinforcement Learning AI & Machine Learning Technology & Innovation Programming & Software Large Language Models Deep Learning Model Training Policy Gradient Proximal Policy Optimization Chain Of Thought Prompting

Mentioned in this video

Software & Apps

SISPO

An algorithm related to PPO, discussed as a popular alternative that makes different choices in clipping strategies to potentially improve stability and performance.

Llama

A family of large language models, mentioned as an example that initially lacked strong thinking capabilities, posing challenges for reinforcement learning due to zero initial rewards.

Quinn

A large language model mentioned as having some initial thinking capability, making it more amenable to reinforcement learning for improving reasoning skills.

Companies

DeepSeek

A large language model mentioned as having some initial thinking capability, making it more amenable to reinforcement learning for improving reasoning skills.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free