Key Moments
Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 13: Intro to Learning
Stanford OnlineWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
While AI can now generate photorealistic virtual humans, the cost to create each one is a staggering $1 million, posing a significant barrier to widespread adoption.
Key Insights
The course transitions from optimal control methods (indirect, direct, dynamic programming, MPC) to learning-based settings where system dynamics are unknown.
Adaptive control focuses on proving the stability of coupled systems (controller, adaptation mechanism, system dynamics) rather than optimizing a performance metric.
System identification uses historical data (state transitions, rollouts) to learn an approximate model of system dynamics, often employing linear regression and least squares minimization.
In Model Reference Adaptive Control (MRAC), performance is measured by the distance to a reference model, and control is achieved through a feedback law with adjustable parameters.
The Lyapunov stability theorem is a primary tool for analyzing the stability of adaptive control systems, requiring a positive-definite Lyapunov function with a negative-definite derivative.
Model Identification Adaptive Control (MIAC) first estimates a model of the system dynamics and then uses this model to improve controller performance.
Transitioning from known to unknown dynamics
This lecture marks a shift from optimal control methods, which assume knowledge of system dynamics, to learning-based control where dynamics are unknown. Previously, the course covered open-loop methods (indirect, direct) for computing optimal trajectories and closed-loop methods (dynamic programming, HJB equations) for computing optimal control policies. Model Predictive Control (MPC) was presented as a hybrid approach. The core assumption underlying these methods was a known, often analytical, representation of system dynamics. The second part of the course relaxes this assumption, focusing on scenarios where system dynamics are not known in advance and must be learned or adapted to.
Strategies for handling system uncertainty
When faced with uncertainty in system dynamics, three main strategies can be employed. First, if unmodeled effects are small (e.g., minor wind gusts on a drone), simple feedback control like PID might suffice. Second, robust control approaches, such as min-max formulations, can be used to design controllers that are resilient to the worst-case scenarios of disturbances. This is effective but can lead to overly conservative behavior. The third, and central focus of this course, is data-driven approaches. These leverage collected data (state transitions, logs of previous behaviors) to learn an approximate model of the system's dynamics. This learned model can then be used to improve control performance.
Two families of learning-based control
Learning-based control methods broadly fall into two categories. The first directly uses collected measurements, episodes, or transitions to improve the controller. The second family uses these measurements to learn an approximate model of the system dynamics, and this learned model then serves as an intermediary to improve the controller. System identification is a key technique within this second family. Today's lecture will cover direct and indirect adaptive control, as well as system identification, laying the groundwork for future topics like imitation learning and reinforcement learning (model-free and model-based).
Paradigms of learning: zero, one, and multiple episodes
Learning processes can be categorized by how data is acquired and utilized. The 'zero episodes' case, often associated with system identification, uses a pre-collected, offline dataset to improve a controller or learn a model. The 'one episode' case, characteristic of online adaptation, involves learning and adaptation happening in real-time while the system is operating. For example, adapting to an unknown payload mass on a drone during flight. The 'multiple episodes' setting is the classical framework for reinforcement learning, where an environment can be reset indefinitely, allowing for learning across numerous interactions and iterations, such as in a game of chess. While boundaries can be blurry (e.g., offline RL fits the zero-episode case), these modalities of experience dictate the suitability of different algorithms.
System identification using linear regression
System identification aims to build a data-driven model of dynamics by learning from a dataset of previous experiments or behaviors. The classical tool for this is linear regression, specifically using least squares minimization. The problem is framed as finding parameters 'theta' in a model like 'y = theta transpose z + noise', where 'y' is the output and 'z' are inputs. Given a dataset 'D' of input-output pairs, the goal is to minimize the squared error between actual and predicted outputs. The solution for 'theta' is derived as (Z transpose Z) inverse * (Z transpose Y), where Z is a matrix of stacked input vectors and Y is a vector of outputs. This approach assumes a linear system model, such as x(t+1) = A*x(t) + B*u(t) + noise, where 'A' and 'B' are the parameters to be estimated. The collected trajectories (state-control pairs) are repurposed as a regression problem mapping current state and control to the next state.
Properties and challenges of system identification estimators
The least squares estimator derived for system identification is known as the Best Linear Unbiased Estimator (BLUE) under certain noise distribution assumptions. If the noise is Gaussian, it's also the Maximum Likelihood Estimator (MLE). The estimator's mean is the true parameter value, implying unbiasedness. Its covariance depends on the variance of the noise and the inverse of a summation related to system inputs over time. For the covariance to approach zero (i.e., for the estimator to converge to the true value), this summation term must grow to infinity, which requires 'persistent excitation' – the system must be probed in non-trivial ways over time. A key challenge is that system identification optimizes model accuracy, which doesn't always correlate with control performance. An estimate might be statistically less accurate but lead to better control if it correctly captures critical dynamics influencing control decisions.
Introduction to adaptive control: online adaptation
Adaptive control falls into the 'one episode' or online adaptation setting, where control is improved in real-time as the system operates. This differs from system identification, which typically uses offline data. Adaptive control can be direct (improving the controller directly) or indirect (learning a model first, then improving the controller). Model Reference Adaptive Control (MRAC) is a direct approach, while Model Identification Adaptive Control (MIAC) is indirect. Classical adaptive control focuses on proving system stability rather than optimizing a specific performance metric. Stability is analyzed using the Lyapunov stability theorem, which requires finding a positive-definite Lyapunov function whose time derivative is negative-definite and radially unbounded. This theorem is applied to analyze the coupled dynamics of the system, the controller, and the adaptation mechanism.
Model Reference Adaptive Control (MRAC) example
MRAC involves four key elements: a plant with unknown parameters, a reference model defining desired behavior, a feedback control law with adjustable parameters, and an adaptation mechanism to update these parameters. An example is presented using a double integrator system (m*x_ddot = u) with an unknown mass 'm'. A reference model is defined with a fictitious state 'xm' that evolves according to a specified differential equation (xm_ddot + Kd*xm_dot + Kp*xm = Kr(t)). If the mass 'm' were known, a controller 'u = mhat * (xm_ddot - 2*lambda*x_tilde_dot - lambda^2*x_tilde)' could achieve zero tracking error (x_tilde = x - xm). When the mass is unknown, an estimate 'mhat' is used, leading to an estimation error 'm_tilde = mhat - m'. The closed-loop dynamics become 'm*s_dot + lambda*s = m_tilde', where 's' is a defined state variable. An adaptation law, 'mhat_dot = -gamma*new*s', adjusts the mass estimate. A Lyapunov candidate function is then used to prove that 's' and consequently the tracking error 'x_tilde' converge to zero, demonstrating the stability of the adaptive system despite the unknown mass.
Mentioned in This Episode
●Software & Apps
●Concepts
Common Questions
Optimal control assumes knowledge of system dynamics to compute optimal trajectories or policies. Learning-based control, conversely, assumes unknown dynamics and uses data to learn and adapt.
Topics
Mentioned in this video
A tool used to analyze the stability properties of autonomous systems by finding a Lyapunov function that satisfies specific conditions.
A function used in the Lyapunov stability theorem that must be positive definite, have a negative semidefinite (or negative definite) derivative along system trajectories, and be radially unbounded.
More from Stanford Online
View all 135 summaries
74 minStanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 16: Fundamentals of RL
80 minStanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 15: Imitation Learning
78 minStanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 17: RL Value-Based Methods
81 minStanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 18: RL Policy Optimization
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free