How is the grading structured for AA203?

The course grade is primarily based on four graded problem sets, each accounting for 20% of the total grade (80% combined). A final exam counts for the remaining 20%. Bonus points up to 5% are available for active participation on Ed, such as meaningfully answering colleagues' questions.

What is the difference between open-loop and closed-loop control?

Open-loop control involves pre-computing a sequence of control actions and executing them without further feedback from the system or environment. Closed-loop control, conversely, continuously measures the system's state and uses this feedback to re-optimize or adjust control actions, making it more robust to disturbances.

How does Model Predictive Control (MPC) bridge open-loop and closed-loop control?

MPC solves an open-loop control problem at each time step, executes a portion of the computed sequence, and then re-computes the control sequence based on new measurements. This iterative process effectively combines the computational efficiency of open-loop solutions with the robustness of closed-loop feedback.

What are the main methodologies for data-driven optimal control covered in the course?

The course will cover imitation learning, where an optimal policy is mimicked from demonstrations, and reinforcement learning, which involves learning through trial and error. Both model-based and model-free approaches within reinforcement learning will be discussed.

What are the three key ingredients of an optimal control problem formulation?

An optimal control problem requires a mathematical description of the system (often differential equations), a set of constraints (e.g., on control inputs or system states), and a performance criterion that defines what constitutes an 'optimal' solution.

How are ordinary differential equations (ODEs) used in optimal control?

ODEs are used to model how a system's state evolves over time based on current states and control inputs. In optimal control, they are viewed not just as mathematical objects but as mechanisms for predicting the future behavior of the system.

What is the role of the performance measure in optimal control?

The performance measure, often a cost functional, quantifies the desirability of a particular control strategy. It typically includes a terminal cost (measuring unhappiness at the final state) and a cumulative cost (integrating costs incurred over time), guiding the selection of the 'best' control.

What is the difference between a local and a global minimum in optimization?

A local minimum is a point where the function's value is lower than at all nearby points. A global minimum is the point with the absolute lowest function value across the entire domain. Not all local minima are global minima.

What are necessary optimality conditions used for in optimization?

Necessary optimality conditions provide criteria that must be met at an optimal point. They act as filters to identify candidate solutions; for example, in unconstrained optimization, the gradient of the function must be zero at a minimum. These conditions extend to optimal control problems.

How does the course handle potential disturbances in control systems?

Disturbances can be modeled in various ways, including deterministically (worst-case scenarios), stochastically (randomly), or by incorporating buffer zones within state constraints. The course will explore techniques like differential games for worst-case disturbances and Markov decision processes for stochastic ones.

Why is optimal control considered an infinite-dimensional optimization problem?

Optimal control involves finding an optimal control function over time, which, in continuous-time formulations, means optimizing an infinite number of values (for every possible time point). This contrasts with classical optimization problems that typically have a finite number of decision variables.

Key Moments

Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 1: Course Overview

Stanford Online

Education6 min read74 min video

Aug 11, 2026|1,581 views|90|3

Stanford Stanford Online Robotics

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Optimal control aims to find the best way to steer systems, blending classical control with machine learning, but it requires understanding complex optimization problems over infinite dimensions.

Key Insights

The course grading breakdown is 80% for four graded problem sets (20% each) and 20% for a final exam, with up to 5% bonus for participation.

Foundational knowledge in multivariable calculus and linear algebra is required; familiarity with optimization, machine learning, and control theory is recommended but not mandatory.

Optimal control problems are defined by three core components: a mathematical model of the system (often ODEs), constraints on states and controls, and a performance criterion (cost function).

The performance criterion typically includes a terminal cost (how unhappy we are at the final time) and an integral term representing cumulative cost per unit time, like control effort or tracking error.

A key distinction is made between open-loop control (pre-computed actions based on time and initial state) and closed-loop control (actions adapted based on current state and environment feedback).

The course will cover both model-based optimal control (assuming a known system model) and data-driven approaches like imitation learning and reinforcement learning (when the model is unknown or learned).

Course structure and grading

The course AA203, Optimal and Learning-Based Control, is taught by Professor Marco Pavone. The grading is structured with 80% allocated to four problem sets, each worth 20%, and 20% for a final exam. Additionally, students can earn up to 5% bonus points for active participation on the course's online platform, such as answering colleagues' questions meaningfully. To help students keep pace, six free late days are provided for assignments, with a maximum of two late days allowed per assignment. All necessary course materials, including lecture slides and supplementary notes, will be available on the course website. Recitations are scheduled for the first four weeks on Fridays to cover foundational tools like Jax and regression models, though attendance is optional for those already proficient.

Prerequisites and self-assessment

Success in AA203 requires a strong comfort level with multivariable calculus and linear algebra, including vector calculus and ordinary differential equations. While not strictly mandatory, prior familiarity with optimization, machine learning, and control theory would significantly ease the learning process. To help students gauge their preparedness, a non-graded 'homework zero' is provided. This assignment serves as a diagnostic tool, allowing students to identify areas where their knowledge might be rusty and need strengthening. The instructor advises that if homework zero proves extremely difficult, students might benefit from taking prerequisite courses before enrolling in AA203.

The motivation and scope of optimal control

Classical control systems aim for stability, tracking, disturbance rejection, and robustness, often using established techniques like PID controllers. However, these methods do not explicitly address performance optimization or how to determine optimal reference signals (planning). Furthermore, they typically assume a known system model, which is often not the case in real-world scenarios, or the model can change over time. This course delves into optimal and learning-based control to address these gaps. It explores how to achieve the 'best' possible system performance, how to control systems when models are unknown or difficult to acquire (learning from data), and how to integrate reference signal generation into the control analysis. The field of optimal control, with roots in the work of Richard Bellman and Pontryagin during the Cold War, seeks to find control strategies that optimize performance beyond mere stability.

Formulations of optimal control: open-loop vs. closed-loop

The course will distinguish between open-loop and closed-loop optimal control. Open-loop control involves computing a sequence of control actions beforehand, assuming these actions will be executed without further information from the environment. While computationally efficient, it is less robust to disturbances or unexpected changes. Closed-loop control, conversely, continuously uses sensor feedback to measure the system's state and re-optimize actions, making it more powerful and robust. Model Predictive Control (MPC) will be presented as a bridge between the two, iteratively solving open-loop problems within a receding horizon framework that incorporates feedback. The first part of the course will focus on methods assuming a known model, covering open-loop control, closed-loop control, and MPC.

Data-driven approaches and learning

The second half of the course shifts to data-driven optimal control, where the system model is not assumed to be known and must be inferred from data. Three broad methodologies will be explored: imitation learning, where an expert's demonstrated optimal policy is mimicked; and reinforcement learning, which learns through trial and error. Reinforcement learning will be discussed in both its model-based and model-free variants. The instructor emphasizes that these techniques are not mutually exclusive and are often hybridized in practice. For instance, imitation learning can be used to bootstrap a policy, which is then refined using reinforcement learning. This holistic overview aims to equip students with the knowledge to select and combine the most appropriate tools for complex robotic and control problems.

Defining the optimal control problem: model, constraints, and performance

At its core, an optimal control problem requires three key ingredients. First, a mathematical model of the system, typically expressed as a set of ordinary differential equations (ODEs) where the future state is a function of the current state and control inputs. ODEs are viewed not just as mathematical objects but as mechanisms for predicting future system behavior. Second, constraints must be considered. These can include initial and final conditions, limitations on state variables (e.g., avoiding certain regions), and bounds on control inputs. A control sequence satisfying all constraints is termed 'admissible.' Third, a performance criterion, or cost function, quantifies what 'optimal' means. This often involves minimizing a combination of a terminal cost (at the final time) and a stage-wise cost integrated over time, which can represent factors like control effort or tracking error. This framework allows for flexibility in defining objectives, such as minimizing time or fuel consumption.

The nature of the optimization challenge

Optimal control problems are inherently challenging because they often involve infinite-dimensional optimization. Unlike classical optimization problems with a finite number of decision variables, optimal control requires optimizing over functions of time (control sequences or policies). This means there are potentially infinite variables to determine. While this sounds daunting, the course will leverage tools from finite-dimensional optimization and introduce concepts like the principle of optimality, derived from the additive structure of the cost function, to manage this complexity. The instructor notes that existence and uniqueness of solutions are not always guaranteed, posing further challenges. Maximization problems can be converted to minimization problems by negating the objective function, although different communities (control theory vs. computer science) use varying terminologies (e.g., cost vs. reward, state vs. state, control vs. action).

Handling disturbances and future topics

Disturbances, such as wind gusts or sensor noise, can be incorporated into optimal control models either directly within the system dynamics (e.g., F depending on a disturbance variable Wk) or indirectly by ensuring robustness. This can involve worst-case modeling (game-theoretical approach) or designing controllers that allow sufficient buffer to avoid hitting constraints even with perturbations. The course will explore various methods for modeling and handling disturbances, including those relevant to differential games and Markov Decision Processes. Famous problems like the 'homicidal chauffeur' and 'lady in the lake' will be used as examples. The lecture concludes by previewing the next steps: a review of classical optimization techniques (unconstrained and constrained nonlinear programming) to build a foundation for tackling the more complex infinite-dimensional optimal control problems.

Mentioned in This Episode

●Software & Apps

●Companies

●Concepts

●People Referenced

Common Questions

Strong familiarity with multivariable calculus, vector calculus, and ordinary differential equations is required. A solid understanding of linear algebra is also necessary. Familiarity with optimization, machine learning, and control theory is beneficial but not strictly required as the class is designed to be self-contained.

Topics

Reinforcement Learning Learning & Education AI & Machine Learning Technology & Innovation Science & Mathematics Dynamic-programming Imitation Learning Mathematical Modeling Robot Autonomy Control Systems Optimal Control Learning-based Control

Mentioned in this video

People

Marco Pavone

The instructor for AA203, Optimal and Learning-Based Control, and a professor at Stanford. He also leads autonomous system research at NVIDIA.

Richard Bellman

A pioneer in optimal control theory in the United States, known for the Bellman equation.

Software & Apps

AA203 Optimal and Learning-Based Control

The course title, focusing on optimal and learning-based control techniques for robotic systems.

Python

The programming language in which coding questions for the problem sets will be implemented.

Jax

A tool that will be covered in the recitations to provide foundational knowledge for the class.

ChatGPT

Mentioned as a tool not to use for Homework Zero to ensure students are testing their own knowledge.

Hamilton-Jacobi-Bellman equations

A continuous-time formulation of dynamic programming used in closed-loop control.

Hamilton-Jacobi-Isaacs equations

A continuous-time formulation of dynamic programming, related to reachability control paradigms.

Companies

NVIDIA

A company where Marco Pavone leads autonomous system research.

Concepts

Bellman Equation

A key concept in decision-making, associated with Richard Bellman.

PID control

A classical control methodology mentioned as a basis for comparison and as a widespread technique.

Markov Decision Processes

A formulation of problems in a stochastic setting, related to discrete-time optimal control.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free