What is the difference between RLHF and RLVR?

RLHF (Reinforcement Learning from Human Feedback) is used when there's no objective ground truth, relying on human preferences (e.g., tone). RLVR (Reinforcement Learning with Verifiable Rewards) applies when answers can be deterministically checked, like in math or coding problems.

How are gradients calculated in Reinforcement Learning for LLMs?

Gradients are calculated by taking the expectation of the rewards multiplied by the gradient of the log probability of the policy. This encourages the model to increase probabilities for positive reward actions and decrease them for negative ones.

Why is on-policy training expensive for LLMs?

On-policy training requires the LLM to generate a response for every update, which is a sequential process (token by token) that cannot be fully parallelized, making it computationally costly compared to evaluation.

What are the main challenges of traditional RL methods for LLMs?

Traditional RL for LLMs faces challenges such as high variance gradients due to non-relative reward functions, lack of explicit token-level credit assignment, and difficulty in preventing policies from deviating too much from initial good models.

How does Proximal Policy Optimization (PPO) address RL challenges?

PPO addresses challenges by maximizing 'advantages' (rewards relative to a baseline) instead of raw rewards, employing a value function for token-level credit, and using clipping and KL divergence to prevent the policy from deviating too much.

What is Group Relative Policy Optimization (GRPO) and how does it differ from PPO?

GRPO aims to maximize advantages without needing a separate value function. Instead, it generates multiple responses for a prompt, calculates advantages by comparing individual rewards to the group's average, and still uses clipping and KL divergence for policy stability.

What is on-policy distillation and why is it useful?

On-policy distillation uses a stronger 'teacher' model to provide dense supervision at the token level for a 'student' model, addressing the lack of token-level credit assignment in other methods. It's efficient, good for parameter constraints, and avoids reward hacking.

How can on-policy distillation avoid needing an external teacher model?

Self-distillation on-policy can be achieved by using the same model as its own teacher. By providing the full context including the correct answer in the prompt, the model can generate target probability distributions for token comparisons.

Key Moments

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

Stanford Online

Education5 min read104 min video

Oct 19, 2026|6,567 views|107|7

Stanford Stanford Online LLM Large Language Models

Save to Pod

Key Moments

TL;DR

Reinforcement learning for LLMs is complex, with methods like PPO and GRPO tackling challenges like noisy gradients and credit assignment, but simpler models might still struggle with nuanced instruction following.

Key Insights

Reinforcement learning (RL) is applied to Large Language Models (LLMs) in the later stages of training, specifically for preference tuning and inference, aiming to align models with human preferences or verify correctness.

The core objective in RL for LLMs is to maximize expected rewards, which is mathematically achieved by calculating gradients that encourage positive reward actions and discourage negative ones.

Policy gradient methods, like REINFORCE, derive a gradient for maximizing rewards, but can suffer from high variance, leading to noisy updates.

Proximal Policy Optimization (PPO) aims to maximize expected advantages (rewards relative to a baseline) and prevent drastic policy deviations using clipped surrogate objectives or KL divergence penalties.

Group Reward Policy Optimization (GRPO) offers an alternative to PPO by avoiding the need for a separate value function, calculating advantages based on reward deviations within a group of generated responses.

On-policy distillation, a newer technique, leverages a teacher model to provide token-level supervision, effectively acting as a form of reinforcement learning without explicit reward models, and is particularly useful for faster student models.

The role of reinforcement learning in LLM training

The lecture begins by situating reinforcement learning (RL) within the LLM training pipeline, positioning it as a crucial step after pre-training and supervised fine-tuning (SFT). While pre-training teaches understanding of text and code, and SFT makes models helpful, RL addresses finer-grained alignment. This includes preference tuning, where RL aligns models with human preferences, and inference, where RL can be used for tasks like mathematical reasoning or code generation. The core idea is to use RL to optimize objectives that are not easily captured by standard cross-entropy loss, such as maximizing a reward function that reflects desired outcomes.

Understanding the RL setup: agent, environment, policy, and reward

The fundamental RL setup involves an agent interacting with an environment. In the context of LLMs, the agent is the LLM itself, the environment is the context of generating text, and the 'state' is the current sequence of tokens. The agent's 'action' is choosing the next token to generate. The 'policy' (often denoted as πθ) is the probability distribution over possible next tokens given the current state, parameterized by the model's weights θ. The goal is to maximize the cumulative reward. Rewards can be challenging to define at the token level, so they are often assigned at the completion or response level. This can be based on human feedback (RLHF) for subjective tasks or verifiable outcomes (RLVR) for objective tasks like math or coding.

The mathematics of policy gradients: maximizing rewards

The primary objective in RL is to maximize expected rewards. Mathematically, this involves finding model parameters θ that maximize the expectation of rewards over trajectories sampled from the policy πθ. To achieve this, gradient ascent is used. The derivation shows that the gradient of the expected reward can be expressed as the expectation of the reward multiplied by the gradient of the log-probability of the policy. Intuitively, this means the gradient encourages actions that lead to higher rewards and discourages those that lead to lower rewards. For LLMs, trajectories are sequences of tokens generated in response to a prompt. The challenge here is that generating these sequences can be computationally expensive, and updating the policy requires generating new sequences.

Challenges with policy gradients: variance and credit assignment

Traditional policy gradient methods, like REINFORCE, face several challenges. High variance in gradients is a major issue; small changes in rewards can lead to large, noisy updates. This can be mitigated by using relative rewards or advantages instead of absolute rewards. Another challenge is credit assignment: rewards are often given at the end of a sequence, making it difficult to determine which specific tokens or actions contributed positively or negatively to the final outcome. For example, in a math problem, a correct final answer might have involved several incorrect intermediate steps, but the standard reward signal would not differentiate this. Furthermore, LLMs are already well-trained, and drastic policy shifts during RL fine-tuning can lead to 'catastrophic forgetting' or 'reward hacking,' where the model exploits flaws in the reward function without genuinely improving the desired behavior. This necessitates constraints to keep the policy close to the original or reference policy.

Proximal Policy Optimization (PPO): balancing exploration and stability

Proximal Policy Optimization (PPO) addresses these challenges by aiming to maximize expected advantages (a measure of how much better an action is than the average) while constraining the policy updates to prevent large deviations from the previous policy. It achieves this through clipped surrogate objectives or by adding a KL divergence penalty to the loss function, which discourages the current policy from straying too far from a reference policy (e.g., the SFT model or a previous iteration of the policy). PPO also introduces a 'value function' that estimates the expected future rewards from a given state, which helps in calculating these advantages more effectively.

Group Reward Policy Optimization (GRPO): a value-function-free approach

GRPO offers an alternative to PPO by avoiding the need for a separate value function. It calculates advantages by comparing the reward of a generated response against the average reward of a batch of responses to the same prompt. This relative measure helps reduce gradient variance. Like PPO, GRPO incorporates mechanisms like clipping or KL divergence penalties to limit policy updates. However, a key limitation of GRPO, as discussed, is that it primarily assigns credit at the completion level, not at the token level, which can be inefficient if intermediate steps are crucial for learning.

On-policy distillation: a supervised approach to RL

On-policy distillation emerges as a promising technique that sidesteps explicit reward functions and value functions, instead leveraging a 'teacher' model to guide a 'student' model. The student model generates responses, and the teacher model, given the same prompt and student-generated tokens, provides a probability distribution for the next token. The loss function then encourages the student's output distribution to match the teacher's. This method provides dense, token-level supervision, is on-policy (meaning it learns from the student's own generations), and is particularly beneficial for faster student models or when computational resources are constrained. It can also be adapted for multi-teacher distillation to cover broader domains or to retain knowledge from previous models.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Concepts

Common Questions

LLM training typically involves three stages: pre-training (learning text and code understanding on vast datasets), supervised fine-tuning (SFT) to make the model useful, and preference alignment using reinforcement learning to incorporate human preferences.

Topics

AI & Machine Learning Programming & Software Science & Mathematics Large Language Models Policy Optimization Deep Learning Model Training Gradient Descent Machine Learning Algorithms Reward Modeling

Mentioned in this video

Concepts

CME 295

The course code for the Stanford lecture on Transformers and LLMs.

Cross-Entropy Loss

The loss function used in pre-training and supervised fine-tuning (SFT) stages of LLM training, penalizing deviations from the expected token.

Bradley-Terry model

A method for expressing loss in a pairwise fashion, used to train a reward model based on human preferences in RLHF.

Reinforcement Learning with Verifiable Rewards

An RL approach used in situations where the correctness of an answer can be deterministically verified, such as math problems or code generation, often applied in inference domains.

Supervised Fine-tuning

A training step where a pre-trained LLM is adjusted to be useful, for example, to provide answers to questions, still using cross-entropy loss but focusing on output tokens.

Reinforcement Learning from Human Feedback

A technique where human feedback is used to measure the quality of responses, typically in subjective situations with no ground truth, leveraging the Bradley-Terry formula.

On-policy distillation

A technique to mitigate the need for token-level credit assignment by having a student model generate completions, then asking a stronger teacher model to evaluate the reasonableness of each token generated.

Reinforcement Learning

A machine learning paradigm where an agent learns to make decisions by performing actions in an environment and receiving rewards, used in LLMs for preference alignment and inference.

Proximal Policy Optimization

A popular reinforcement learning algorithm that aims to maximize an objective function while maintaining proximity to previous policies, utilizing advantages and a value function.

Generalized Advantage Estimation

A technique mentioned for estimating advantages in RL, which considers how future values relate to a specific token, making the advantage calculation more complex than a simple value difference.

Direct Preference Optimization

An approach mentioned in the context of supervised methods for RL objectives, specifically linking it to the beta coefficient in KL divergence penalties for staying close to a reference policy.

Kullback-Leibler (KL) divergence

A measure of how one probability distribution diverges from a second, expected probability distribution, used in RL to penalize policies that deviate too much from a reference policy.

Group Relative Policy Optimization

A reinforcement learning algorithm that aims to maximize advantages without needing a separate value function, instead comparing generated responses within a group to calculate relative advantages.

Organizations

Frontier Lab

A hypothetical organization used to illustrate how on-policy distillation might be applied to faster, smaller models (e.g., 'Flash' series) while larger models ('Pro' series) might use GRPO.

Software & Apps

GLM 5

A model mentioned as an example where on-policy distillation can be used for calibration to avoid losing previously learned knowledge, effectively distilling knowledge from itself.

Companies

Thinking Machines

A company mentioned for publishing a blog post with illustrative ablation studies on on-policy distillation, showing its benefits under computational constraints.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free