Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 10: Reachibility Analysis

In the infinite horizon setting, the Bellman equation becomes a fixed-point equation. Instead of a finite sum of rewards, it considers an infinite sum (or average) of rewards, and the optimal value function satisfies this fixed-point equation.

The HJB equation is the continuous-time counterpart to the Bellman equation in dynamic programming. It's a partial differential equation used to find the optimal control policy for systems evolving in continuous time.

Differential games involve optimal control problems with multiple players whose actions are governed by differential dynamics. They are useful for modeling systems with adversarial disturbances, like collision avoidance, where robustness is critical.

In a minimax formulation, player one seeks to maximize their outcome while assuming player two will act to minimize it. A maximin formulation is the inverse. The choice depends on which player has the informational advantage, which in this lecture is given to the disturbance player.

The HJB equation is derived by applying the principle of optimality to continuous-time systems, approximating small time intervals with first-order Taylor expansions, and solving a partial differential equation that incorporates the system dynamics and cost functions.

The continuous-time LQR problem involves linear system dynamics described by differential equations and a quadratic cost function represented by an integral. Its solution still relies on dynamic programming principles, leading to a linear feedback control law.

A backward reachable set is the set of all initial states from which a system is guaranteed to reach a target set by a specific final time, considering potential adversarial actions or disturbances.

The Hamilton-Jacobi-Isaacs equation is the solution to differential games (problems with adversarial players), extending the HJB equation by incorporating a minimax or maximin structure due to the conflicting objectives of the players.

Non-anticipatory strategies mean that a player (e.g., the disturbance) can only react to the current state and past actions of the other player, not predict future actions. This assumption simplifies the problem but gives a slight advantage to the reacting player.