Key Moments
Stanford CS229 Machine Learning | Spring 2026 | Lecture 4: Exponential Family, GLMs Classification
Stanford OnlineWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Exponential families provide a unified framework for diverse probability distributions, allowing a single set of learning and inference algorithms to apply across many models, from simple coin flips to complex AI language generation.
Key Insights
The exponential family's general form unifies many statistical models, allowing inference and learning algorithms to be applied universally once a distribution is expressed in this form.
The Bernoulli and Gaussian distributions, commonly encountered in linear models, both fit within the exponential family, demonstrating its broad applicability.
The log-partition function within the exponential family's form, when differentiated, directly yields the expected value of the sufficient statistic, providing a direct link between model parameters and observable quantities.
Generalized Linear Models (GLMs) combine a linear predictor with an error model from the exponential family, using a link function to connect them, which forms the basis for many machine learning tasks.
Softmax, a key component in multiclass classification and modern AI like large language models, is a specific application of the exponential family, enabling the selection of one token out of many possibilities.
Label smoothing, a technique used with cross-entropy loss in softmax, acts as a form of regularization by preventing overfitting and making models more robust to noisy labels.
The power of the exponential family for unifying models
The exponential family is a fundamental concept in statistics that provides a unified framework for a vast array of probability distributions. Its significance lies in its ability to express diverse models, from regression and classification to count data, in a single, standardized mathematical form. This standardization is crucial because once a distribution adheres to this form, a consistent set of algorithms for inference (e.g., computing means and variances) and learning (e.g., parameter estimation) becomes applicable. This means that the techniques learned for simple models like linear regression or logistic regression can be readily generalized to more complex distributions, saving considerable effort in algorithm development and understanding. The core idea is that by recognizing a distribution as belonging to the exponential family, we gain access to 'free' inference and learning capabilities that are common across the entire family.
Deconstructing the exponential family's functional form
The general form of a probability distribution within the exponential family is given by P(y | eta) = exp(eta^T T(y) - A(eta)) * B(y). Here, 'eta' represents the natural parameters of the distribution, which are learned from data. 'T(y)' is known as the sufficient statistic, capturing what we measure about the variable 'y'. 'B(y)' is the base measure, and crucially, 'A(eta)' is the log-partition function, which acts as a normalization constant. This log-partition function is particularly important because it implicitly defines the range of possible values for 'y' and, through its derivatives, reveals key properties of the distribution. The structure ensures that for every parameter in the distribution, there's a corresponding natural parameter in 'eta', and the dimensionality of 'T(y)' matches that of 'eta'.
Bernoulli and Gaussian distributions as examples
The lecture demonstrates how common distributions, such as the Bernoulli and Gaussian, fit into the exponential family framework. For the Bernoulli distribution, used in binary classification (like a coin flip), the probability of observing a '1' is modeled. By applying logarithmic transformations and algebraic manipulation, the Bernoulli probability mass function can be rewritten into the standard exponential family form. Similarly, the Gaussian (normal) distribution, often used for modeling continuous errors in least squares, also conforms to this structure. These examples serve as validation, showing that previously studied models are special cases of this broader family, reinforcing the utility of the exponential family for unifying machine learning concepts.
Inference and learning through derivatives of the log-partition function
A powerful property of the exponential family is that its statistical moments can be derived by taking derivatives of the log-partition function, A(eta). Specifically, the expected value of the sufficient statistic T(y) is equal to the first derivative of A(eta) with respect to eta. This means that once the distribution is in the correct form, calculating the mean of the data is simplified. Furthermore, the variance of T(y) can be found by taking the second derivative of A(eta). This relationship provides an elegant and general way to perform inference for any distribution within the exponential family, consolidating many separate derivations into a single, unified procedure.
Generalized Linear Models (GLMs) connect linear predictors to error distributions
Generalized Linear Models (GLMs) build upon the exponential family by linking a linear combination of input features (theta^T * x) to the expected value of the target variable through a 'link function'. The linear predictor (theta^T * x) generates a raw score, which is then transformed by the link function to produce the natural parameter 'eta' for the chosen exponential family distribution. This structure allows for modeling a wide variety of data types. For instance, if Y is binary, the logistic link function connects the linear predictor to the Bernoulli distribution, resulting in logistic regression. If Y is continuous, a linear link connects to the Gaussian distribution for linear regression. The choice of error distribution (from the exponential family) and the link function determines the specific GLM and its behavior.
Softmax for multiclass classification and modern AI
Softmax is a generalization of the sigmoid function used in logistic regression, enabling classification among more than two classes. It's a prime example of an exponential family model. For an input 'x' and learned parameters 'theta_j' for each class 'j', the softmax function computes the probability of 'x' belonging to class 'j' by exponentiating the dot product (theta_j^T * x) and normalizing across all classes. This process is fundamental to tasks like image classification and, crucially, to modern large language models, where it's used to predict the probability of the next word or token from a vast vocabulary. The underlying principle is to assign a probability distribution over all possible discrete outcomes.
Cross-entropy loss and label smoothing for robust training
Training models with softmax typically uses cross-entropy loss, which measures the difference between the predicted probability distribution and the true (one-hot encoded) label. The derivation shows that maximizing the log-likelihood of the data under the multinomial distribution (which underlies softmax) leads directly to the cross-entropy loss. Label smoothing is a regularization technique often applied alongside cross-entropy. Instead of assigning a probability of 1 to the true class and 0 to others, it slightly 'smooths' the true distribution by assigning a small probability mass (epsilon) to all classes. This helps prevent overfitting by discouraging the model from becoming overly confident in its predictions and improves robustness to potential errors in the training labels.
Mentioned in This Episode
●Software & Apps
●Companies
●Concepts
Common Questions
The exponential family is a broad class of probability distributions that share a common functional form. This structure allows for a unified approach to modeling various data types, simplifying inference and learning procedures across different models like logistic regression and Gaussian distributions.
Topics
Mentioned in this video
Mentioned as a framework that existed in the past, contrasting with the manual coding required in earlier days.
A key component for multiclass classification, used in AI models for next-token prediction and foundational to modern AI architectures.
Mentioned as an example of an AI model with a specific dimensionality (768) relevant to the discussion on high-dimensional data separation.
Mentioned as a common library for implementing softmax and neural network functionalities, highlighting practical application.
Used as a familiar baseline model that is generalized by the exponential family and softmax for multiclass classification.
Used as a foundational example of a distribution within the exponential family, applicable to binary outcomes like coin flips.
Another example distribution that fits into the exponential family framework, relevant for understanding errors in least squares.
The architecture underlying modern AI, which utilizes attention mechanisms, itself related to multiclass classification concepts discussed.
More from Stanford Online
View all 106 summaries
77 minStanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA
76 minStanford CS229 Machine Learning | Spring 2026 | Lecture 12: Representation Learning
74 minStanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient
81 minStanford CS229 Machine Learning | Spring 2026 | Lecture 10: GMM (EM), PCA
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free