Key Moments
Stanford CS229 Machine Learning | Spring 2026 | Lecture 5: Gaussian Discriminant Analysis
Stanford OnlineWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Generative models like GDA learn data distributions for each class, enabling AI advancements like GPT. However, classic GDA with shared covariance simplifies to a linear decision boundary, similar to logistic regression, undermining its generative power.
Key Insights
Generative models, the foundation of systems like GPT, learn data representations, a shift from earlier discriminative models focused solely on decision boundaries.
Gaussian Discriminant Analysis (GDA) models data within each class as a Gaussian distribution, assuming shared covariance between classes for a simplified linear decision boundary.
While GDA with shared covariance simplifies to a linear decision boundary (like logistic regression), Quadratic Discriminant Analysis (QDA) allows for non-linear, quadratic boundaries by not sharing covariance matrices.
Naive Bayes, a generative model for discrete features, simplifies the problem by assuming conditional independence of features given the class, drastically reducing the number of parameters from exponential to linear.
Maximum Likelihood Estimation (MLE) for both GDA and Naive Bayes allows for closed-form solutions, making parameter estimation computationally cheap and efficient, unlike gradient descent methods.
The progression from GDA to Naive Bayes showcases how different generative assumptions (continuous Gaussians vs. discrete conditionally independent features) can lead to models with varying complexity and decision boundaries.
The rise of generative models and their distinction from discriminative approaches
The lecture introduces generative models, highlighting their fundamental role in modern AI like GPT. Unlike discriminative models, which focus on learning a boundary to separate classes, generative models aim to learn the underlying data distribution for each class. This approach allows the model to 'understand' how the data for each class is generated. Initially, generative models were less favored due to performance limitations, but their ability to learn rich data representations has made them central to recent AI breakthroughs. The core idea is to model P(X|Y) (the data distribution given the class) rather than just P(Y|X) (the class probability given the data), which is the focus of discriminative models.
Gaussian Discriminant Analysis (GDA) for continuous data
GDA is presented as the first generative model discussed, focusing on continuous features. It assumes that the data distribution for each class follows a Gaussian (normal) distribution. In its simplified form, GDA assumes that all classes share the same covariance matrix but have different mean vectors. The mean vector represents the center of the distribution for that class, while the covariance matrix describes the shape, size, and orientation of the distribution. For 1D data, this is a standard Gaussian curve defined by its mean and variance. In higher dimensions (e.g., 2D), the Gaussian is represented by a mean vector and a covariance matrix, visualized as elliptical contours. The covariance matrix is crucial as it captures correlations between features and the spread of the data, allowing for richer representations than just assuming independence.
Parameter estimation for GDA using Maximum Likelihood Estimation (MLE)
The parameters for GDA – the class prior probability (how likely each class is), the mean vectors for each class, and the shared covariance matrix – can be estimated using Maximum Likelihood Estimation (MLE). The lecture demonstrates that for GDA, these MLE estimates have closed-form solutions, meaning they can be calculated directly without iterative optimization methods like gradient descent. The class prior is estimated by the fraction of data points belonging to that class. The mean vectors are simply the average of the data points within each class. The shared covariance matrix is calculated by pooling the data from all classes, computing the deviation of each data point from its class mean, and then averaging these deviations. This efficient, direct estimation is a significant advantage.
The decision boundary of GDA and its relation to logistic regression
A key finding is that when GDA assumes a shared covariance matrix across classes, its decision boundary becomes linear, forming a hyperplane. This happens because the quadratic terms involving the covariance matrix cancel out when equating the probability distributions of two classes. This linear decision boundary means that, in this specific configuration, GDA produces the same classification outcome as logistic regression, a discriminative model. While the underlying generative assumptions are different, the resulting linear separation is identical. This suggests that assuming a shared covariance might be an oversimplification that sacrifices the full generative modeling potential for a linear separation that a simpler discriminative model could achieve.
Quadratic Discriminant Analysis (QDA) for non-linear decision boundaries
To overcome the limitation of a linear decision boundary in GDA, Quadratic Discriminant Analysis (QDA) is introduced. In QDA, each class is allowed to have its own distinct covariance matrix. This relaxation means the quadratic terms involving the covariance matrices do not cancel out, leading to quadratic decision boundaries. These quadratic boundaries are more flexible and can capture more complex relationships in the data, unlike the strictly linear boundaries of GDA with shared covariance or logistic regression. Estimating parameters for QDA involves calculating separate means and covariance matrices for each class, which increases the number of parameters compared to GDA.
Naive Bayes for discrete features and its simplifying assumption
When dealing with discrete features, such as words in text classification, Naive Bayes is presented as a simple yet effective generative model. The core challenge with discrete features, especially in high dimensions (like a large vocabulary), is the exponential explosion of possible data combinations. Naive Bayes addresses this by making a strong simplifying assumption: it assumes that all features are conditionally independent given the class. This 'naive' assumption means that, for example, the presence of one word in an email doesn't influence the probability of another word appearing, given that we know whether the email is spam or not. This assumption drastically reduces the number of parameters needed, from an exponential number to a linear one, making the model computationally tractable.
Parameter estimation and efficiency of Naive Bayes
Similar to GDA, Naive Bayes also benefits from simple and efficient parameter estimation using MLE. The class prior is the fraction of data in each class, and the conditional probabilities of each feature given a class are estimated by counting occurrences within each class. For text classification with a bag-of-words model, this involves calculating the probability of each word appearing in spam emails versus non-spam emails. Laplace smoothing (adding a small count, like one, to all feature counts) is often used to prevent zero probabilities for unseen words, making the model more robust. The entire process is computationally very cheap, both for training and for making predictions, making it suitable for large datasets.
The broader impact and evolution of generative models in AI
The lecture concludes by emphasizing that while classic models like GDA and Naive Bayes have clear mathematical derivations and efficient estimators, the underlying principle of modeling data generation has driven significant progress in modern AI. Generative models are now at the forefront of advancements in image generation (diffusion models, GANs) and text generation (LLMs like GPT). These advanced models, while more complex, build on the fundamental idea of understanding and replicating data distributions. The lecture notes that many generative models, under certain conditions or when using specific feature representations, can indeed lead to models that behave similarly to logistic regression, but the true power and excitement in recent years have come from exploring the rich representations and capabilities unlocked by generative approaches, especially in unsupervised and self-supervised learning.
Mentioned in This Episode
●Software & Apps
●Concepts
Common Questions
Discriminative models learn the decision boundary between classes directly (P(Y|X)), while generative models learn the distribution of data for each class (P(X|Y)) and then use Bayes' rule for classification. Generative models aim to understand how data is constructed.
Topics
Mentioned in this video
Mentioned as an example of generative pre-trained transformers, highlighting the importance of generative models in modern AI.
Mentioned as an example of generative pre-trained transformers, highlighting the importance of generative models in modern AI.
Mentioned as an AI tool used to rewrite and generate plots for the lecture's visuals. The speaker notes some of the generated graphics and hopes they are okay.
Mentioned as pre-trained models preceding GPT, used for encoding text and then performing classification.
Cited as an example of a diffusion model that emerged around five to six years prior to the lecture.
The core distribution discussed in the lecture, fundamental to Gaussian Discriminant Analysis. Its properties in 1D and 2D are explained.
The main topic of the lecture, presented as the first example of a generative algorithm that learns data representations for discrimination.
Mentioned as the distribution used in logistic regression for modeling P(Y|X).
Discussed as generative and discriminative models popular around 2016, still valuable and cool.
The method used to estimate parameters for Gaussian Discriminant Analysis, particularly highlighting that estimation can be done quickly without extensive learning.
Used as the fundamental rule for classification in generative models, relating P(Y|X) to P(X|Y) and P(Y).
Introduced as a variant of GDA where covariance matrices are not shared, leading to quadratic decision boundaries.
Presented as a modern AI application that iteratively denoises from Gaussians, used for generating images and text.
Mentioned as generative models in modern AI, with examples like GPT.
Discussed in relation to exponential family models and logistic regression, highlighting their common linear component.
Mentioned as the underlying mathematical framework for diffusion models, though often replaced by tricks in practice.
More from Stanford Online
View all 106 summaries
77 minStanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA
76 minStanford CS229 Machine Learning | Spring 2026 | Lecture 12: Representation Learning
74 minStanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient
81 minStanford CS229 Machine Learning | Spring 2026 | Lecture 10: GMM (EM), PCA
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free