What is a Markov Decision Process (MDP)?

An MDP is a mathematical framework used to model decision-making in situations where outcomes are partly random and partly under the control of a decision-maker. It involves states, actions, transition dynamics, and rewards.

What are the key components of an MDP?

The key components of an MDP are the set of states (S), the set of actions (A), the transition dynamics (P), the reward function (R), and the discount factor (gamma).

How is the reward function defined in RL?

A reward function assigns a scalar value indicating how good a particular state is. It can be a function of the state alone, or of the state and action, or even state, action, and next state. Its primary purpose is to guide the agent's learning.

What is reward shaping and why is it used?

Reward shaping involves modifying the reward function to provide intermediate signals, often to make learning easier or faster. While it can guide the agent, it can also lead to unintended consequences if not carefully designed.

What is the discount factor in RL?

The discount factor (gamma) is a value between 0 and 1 that determines the present value of future rewards. A lower gamma makes the agent more myopic, while a higher gamma encourages long-term planning.

What is a policy in reinforcement learning?

A policy is a strategy or mapping that an agent uses to decide which action to take in a given state. It can be deterministic (always choosing the same action) or stochastic (choosing actions based on a probability distribution).

What is a value function in RL?

A value function estimates the expected total future reward (return) an agent can achieve starting from a particular state, given a specific policy. It helps in evaluating the 'goodness' of states.

What is the Bellman Equation?

The Bellman Equation is a recursive relation used in dynamic programming and RL that breaks down the value of a state into the immediate reward plus the discounted value of future states. It's fundamental for solving MDPs.

What is the Policy Gradient algorithm?

Policy Gradient, also known as REINFORCE, is an algorithm that directly optimizes the policy's parameters by estimating the gradient of the expected return with respect to those parameters.

How does the Policy Gradient algorithm estimate gradients?

It uses a technique that rewrites the gradient calculation to involve the gradient of the log probability of actions, allowing for estimation through sampling from the policy's distribution.

Why are stochastic policies useful in training?

Stochastic policies introduce continuity into the training process, allowing for smoother updates to parameters compared to deterministic policies which can have hard switches between actions. This is crucial for gradient-based optimization.

Key Moments

Stanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA

Stanford Online

Education7 min read77 min video

Jul 31, 2026|364 views|7|1

Stanford Stanford Online Machine Learning Artificial Intelligence AI

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Reinforcement learning allows AI to learn through trial-and-error using rewards, not explicit labels, making it ideal for sequential decision-making tasks like robotics and LLMs.

Key Insights

Reinforcement learning (RL) tackles sequential decision-making problems where actions have future ramifications, distinguishing it from simple prediction tasks.

RL operates with minimal supervision, learning from scalar rewards that indicate 'good' or 'bad' outcomes rather than explicit correct actions.

A Markov Decision Process (MDP) models the RL environment with states (S), actions (A), transition dynamics (P), and rewards (R), forming the basis for interaction.

The goal in RL is to maximize expected return (discounted sum of rewards), often incorporating a discount factor (gamma) to prioritize earlier rewards.

Policy gradient methods, like the REINFORCE algorithm, are used to train stochastic policies by estimating gradients of the expected return with respect to policy parameters.

The core challenge in policy gradient is estimating gradients when the policy parameters only influence the sampling distribution, not the reward function directly, solved by the score function trick (nabla_theta log p).

The essence of sequential decision-making in reinforcement learning

Reinforcement learning (RL) is introduced as a technique for solving sequential decision-making problems. Unlike tasks that involve a single prediction, RL requires agents to make a series of decisions where each action can influence future outcomes. This complexity means that a locally optimal decision might not be globally optimal, necessitating consideration of downstream consequences. Furthermore, RL often involves balancing the trade-off between exploration (gathering information) and exploitation (using existing knowledge for immediate gain). While this trade-off is crucial, the lecture focuses primarily on the sequential aspect and the long-term ramifications of decisions, particularly in contexts where explicit exploration strategies might not be heavily emphasized.

Learning from rewards without direct supervision

A key characteristic of reinforcement learning is its reliance on minimal supervision. Instead of receiving explicit labels for correct actions, RL agents learn from scalar rewards. A reward signal indicates whether a sequence of actions was good or bad, but it does not specify the optimal action itself. This 'learning from reward' paradigm, as opposed to learning from labels, necessitates a trial-and-error approach. Agents actively collect data by taking actions, observing the outcomes, and using the reward feedback to reinforce beneficial actions and penalize detrimental ones. This forms a continuous loop of data collection and training.

The Markov Decision Process (MDP) framework

To formalize the interaction with an environment, reinforcement learning utilizes the Markov Decision Process (MDP). An MDP is defined by several key components: a set of states (S), representing all possible configurations of the environment; a set of actions (A) that the agent can take; transition dynamics (P), a probabilistic function describing the likelihood of moving to a next state (s') given a current state (s) and action (a); and a reward function (R), which assigns a scalar value to states or state-action pairs, indicating their desirability. The 'Markov property' is central, stating that the future state depends only on the current state and action, not on the history of previous states and actions. This allows for a simplified model of the world, where only the present state is relevant for future predictions. The state space (S) can be discrete or continuous, and similarly, the action space (A) can be discrete or continuous. Transition dynamics can be deterministic (a specific next state is always reached) or stochastic (a probability distribution over next states). Rewards can be defined solely on states, or on state-action pairs, or even on state-action-next state tuples, though the simpler forms are often used for clarity. For instance, in a robot navigation task, the state could be the robot's position, actions could be 'move left' or 'move right', transition dynamics would model movement accuracy, and rewards would be given for reaching a goal state.

Defining the objective: Return and Discount Factor

The overall goal in RL is to maximize the cumulative reward over a sequence of actions, known as the return or payoff. For a given trajectory (sequence of states and actions), the return is typically calculated as the sum of rewards received at each time step. To account for the temporal nature of decisions and potential infinite horizons, a discount factor (gamma), typically a value between 0 and 1, is introduced. This discounts future rewards, making rewards obtained earlier more valuable than those obtained later. The discounted return for a trajectory is then the sum of gamma^t * R_t at each time step t. This discounting not only prioritizes immediate rewards but also ensures that the total return remains bounded, even for infinite trajectories, which simplifies mathematical analysis. The choice of gamma reflects the agent's preference for immediacy versus long-term gains.

Policies: The agent's strategy for decision-making

The output of an RL agent is a policy, which dictates how the agent should act in any given state. A policy (pi) can be viewed as a mapping from states to actions. In the context of MDPs and the Markov property, the optimal policy only needs to consider the current state, not the entire history. Policies can be deterministic, specifying a single action for each state, or stochastic, specifying a probability distribution over actions for each state. While it's often proven that a deterministic optimal policy exists, stochastic policies are frequently used in practice, especially during training, as they can facilitate exploration and provide smoother gradients for optimization algorithms. The goal of RL is to find an optimal policy (pi*) that maximizes the expected return from any given state.

Value functions: Quantifying the goodness of states and policies

Value functions are used to quantify the expected return of a policy or the maximum possible return from a state. The state-value function V_pi(s) represents the expected total return an agent can expect to receive starting from state 's' and following policy 'pi'. This function implicitly captures the long-term consequences of being in a particular state under a specific policy. The optimal state-value function, V*(s), represents the maximum expected return achievable from state 's', assuming the agent follows the optimal policy from that point onwards. These value functions are crucial for understanding and evaluating different policies and are often computed using methods like the Bellman equation, which provides a recursive relationship to calculate these values.

The Bellman equation: A recursive framework for value computation

The Bellman equation provides a fundamental recursive relationship for computing value functions. For a given policy pi, the value of a state V_pi(s) can be expressed in terms of the immediate reward and the discounted value of the next state. Specifically, V_pi(s) is equal to the expected immediate reward plus the discounted expected value of the next state, where the expectation is taken over possible next states (s') based on the transition dynamics P(s'|s, a) and actions (a) sampled from the policy pi(a|s). This recursive structure allows for iterative computation of value functions. For example, the geometric series 1 + gamma + gamma^2 + ... can be solved using a Bellman-like recursion: V = 1 + gamma*V, leading to V = 1/(1-gamma). In RL, the Bellman equation is applied to systems of linear equations, where each state's value depends on the values of other states, enabling solutions through linear system solvers.

Policy gradient and the REINFORCE algorithm

Policy gradient methods, such as the REINFORCE algorithm, are central to training reinforcement learning agents, particularly for large language models where direct value function optimization can be challenging. These algorithms aim to directly optimize the policy parameters (theta) to maximize the expected return. The core idea is to compute the gradient of the expected return with respect to the policy parameters and use this gradient to update the parameters via gradient ascent. A key challenge is that the policy parameters (theta) only influence the probability distribution from which actions are sampled, not the reward function itself. This is addressed using the 'score function trick', which reformulates the gradient as an expectation involving the gradient of the log probability of the sampled actions. This allows for gradient estimation through sampling, even when the transition dynamics of the environment are unknown.

Mentioned in This Episode

●Software & Apps

●Companies

●Concepts

Common Questions

Reinforcement learning is a technique for sequential decision-making where an agent learns by taking actions in an environment to maximize a reward signal. Unlike supervised learning, it doesn't rely on labeled data but learns through trial and error.

Topics

Reinforcement Learning AI & Machine Learning Technology & Innovation Science & Mathematics Sequential Decision Making Value Functions Expected Payoff Stochastic Policies

Mentioned in this video

Companies

Amazon

Mentioned as an example of sequential decision-making in making an order.

Concepts

Markov decision process

The basic modeling framework for applying reinforcement learning, describing how decisions affect the environment and future states.

Bellman Equation

A fundamental concept in reinforcement learning used to write recursions to reason about the world and solve complex problems by getting rid of the sequential aspect.

Policy Gradient

A core algorithm for reinforcement learning, particularly useful for robotics and large language models, that directly optimizes the policy.

Software & Apps

REINFORCE

An alternative name for the Policy Gradient algorithm, historically significant in reinforcement learning.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free