Key Moments
Stanford CS229 Machine Learning | Spring 2026 | Lecture 10: GMM (EM), PCA
Stanford OnlineWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
The EM algorithm uses a clever trick with Jensen's inequality to find parameters for latent variable models like GMMs, but PCA is often misapplied due to assumptions about data separation.
Key Insights
The EM algorithm constructs a lower bound (ELBO) for the log-likelihood by introducing a surrogate function (LT) that is tangent to the original function at the current parameter estimate and lies below it.
For Gaussian Mixture Models (GMMs), the EM algorithm recovers an ad hoc clustering method that uses soft assignments (probabilities of belonging to each cluster) instead of hard assignments, fitting the mean, covariance, and proportion for each component.
The E-step in EM involves calculating the conditional probability of the latent variable Z given the observed data X and current parameters (calculating QIs), essentially assigning points probabilistically to clusters.
The M-step in EM maximizes the surrogate function LT with respect to the model parameters (mean, covariance, proportions) to find the next parameter estimate.
PCA seeks to find directions of maximal variance in the data by centering the data and finding the eigenvectors of the covariance matrix, which represent the principal components.
PCA is sensitive to the scaling of features; data should ideally be standardized or normalized to prevent features with larger scales from dominating the principal components.
The EM algorithm for GMMs: A principled approach to clustering
The lecture begins by revisiting the Gaussian Mixture Model (GMM) and its relation to K-means, highlighting that GMMs use soft assignments (probabilities) for data points belonging to clusters, unlike K-means' hard assignments. The Expectation-Maximization (EM) algorithm provides a principled way to fit GMMs. It iteratively refines estimates for cluster means (μ), covariances (Σ), and proportions (φ). The core idea of EM is to optimize a surrogate function that is guaranteed to be a lower bound of the true log-likelihood, and iteratively optimizing this surrogate leads to an increase in the log-likelihood. This approach is particularly useful for models with latent variables, such as the cluster assignment (Z) in GMMs, which are not directly observed.
Jensen's inequality and the construction of the surrogate function (ELBO)
A key mathematical tool enabling EM is Jensen's inequality, particularly for concave functions like the logarithm. The algorithm cleverly introduces an auxiliary distribution Q(Z) over the latent variables. By rewriting the log-likelihood as an expectation over Q(Z) and then applying Jensen's inequality, a lower bound (the Evidence Lower Bound or ELBO) is derived. This lower bound is easier to optimize. The 'tightness' of this lower bound is achieved when Q(Z) is chosen to be the conditional distribution P(Z|X, θ), effectively making the expectation an equality. This specific choice of Q(Z) ensures that the surrogate function LT is tangent to the original log-likelihood function at the current parameter estimate θ_t, and thus, maximizing LT will improve the original log-likelihood.
The E-step and M-step in practice
The EM algorithm proceeds in two steps: the E-step (Expectation) and the M-step (Maximization). In the E-step, given the current parameters θ_t, the algorithm computes the expected values of the latent variables, essentially calculating the probability of each data point belonging to each cluster (Q_i(Z)). This is akin to soft assignment. This step essentially formalizes Bayes' rule to determine the posterior probability of a data point belonging to a specific cluster, considering the current model parameters. The M-step then uses these computed probabilities to re-estimate the model parameters (μ, Σ, φ) by maximizing the expected complete-data log-likelihood, which is derived from the ELBO. This process is repeated until convergence.
Recovering ad hoc methods and convergence properties
Crucially, for GMMs, the EM algorithm recovers the intuitive, ad hoc clustering steps that might have been derived manually. The soft assignments in the E-step are derived using Bayes' rule, comparing the likelihood of a data point belonging to each Gaussian component, weighted by the component's proportion. The M-step involves computing derivatives of the objective function with respect to the parameters (μ, Σ, φ) and setting them to zero to find the maximum. The algorithm is guaranteed to monotonically increase the log-likelihood, ensuring convergence. However, it only guarantees convergence to a local optimum, not necessarily the global optimum, meaning the initial choice of parameters can influence the final result.
Introduction to Principal Component Analysis (PCA)
The lecture then transitions to Principal Component Analysis (PCA), a fundamental technique for dimensionality reduction. PCA aims to find a new set of orthogonal axes (principal components) that capture the maximum variance in the data. The first principal component captures the most variance, the second captures the next most (orthogonal to the first), and so on. This is useful for visualizing high-dimensional data, reducing noise, and improving the efficiency of other machine learning algorithms by working with fewer, more informative features.
PCA: Centering, variance, and the covariance matrix
To perform PCA, the data is first centered by subtracting the mean of each feature. This ensures that the analysis focuses on the variance rather than the location of the data. The goal is to find a direction (a unit vector U1) such that when data points are projected onto this direction, the variance of these projected points is maximized. This is equivalent to minimizing the perpendicular distance (residual) from the data points to the line defined by U1. Mathematically, this involves finding the eigenvectors of the data's covariance matrix. The eigenvectors represent the directions of maximum variance, and their corresponding eigenvalues indicate the amount of variance along those directions.
Eigenvalue decomposition and selecting principal components
The core mathematical operation in PCA is the eigenvalue decomposition of the covariance matrix. The eigenvectors of the covariance matrix are the principal components, and the eigenvalues represent the variance explained by each component. By convention, eigenvalues are sorted in descending order, so the first eigenvector corresponds to the direction of greatest variance. For dimensionality reduction, one selects the top 'k' eigenvectors (principal components) that explain a sufficient amount of the total variance, effectively projecting the data onto a lower-dimensional subspace.
Caveats and practical considerations for PCA
Several practical considerations are highlighted. PCA is sensitive to the scale of features; if features have vastly different scales (e.g., feet vs. inches), the feature with the larger scale might dominate the principal components. Therefore, it's often recommended to standardize or normalize the data before applying PCA. Another crucial caveat is that PCA assumes that the principal components are well-separated. If eigenvalues are close, the corresponding eigenvectors (principal components) might be unstable, leading to different results on different runs or subsets of data, which can be problematic for subsequent machine learning tasks. PCA also doesn't inherently handle missing data; imputation or data deletion is typically required beforehand.
Mentioned in This Episode
●Software & Apps
●Concepts
Common Questions
K-Means uses hard assignments, meaning each data point belongs to a single cluster. Gaussian Mixture Models (GMM) use soft assignments, where a data point can belong to multiple clusters with varying probabilities, allowing for more nuanced clustering.
Topics
Mentioned in this video
K-means is an unsupervised learning algorithm used for clustering. It partitions data points into K distinct clusters based on their proximity to the cluster centroids, using a hard assignment approach where each point belongs to exactly one cluster.
Python is a high-level, interpreted, general-purpose programming language. It is widely used in machine learning and data science for its extensive libraries and ease of use.
Jensen's inequality relates the value of a concave (or convex) function of a random variable to the random variable's expected value. For a concave function, the expected value of the function is less than or equal to the function of the expected value.
Bayes' rule is a fundamental theorem in probability theory that describes the probability of an event, based on prior knowledge of conditions that might be related to the event. It is used to update beliefs in light of new evidence.
More from Stanford Online
View all 106 summaries
77 minStanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA
74 minStanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient
79 minStanford CS229 Machine Learning | Spring 2026 | Lecture 20: GMM (EM), PCA
61 minStanford CS229 Machine Learning | Spring 2026 | Lecture 13: LLMs, Next-Word Prediction Loss
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free