How does maximum likelihood relate to linear regression?

The maximum likelihood principle provides a probabilistic justification for least squares. By assuming a probabilistic model for the data with Gaussian noise, maximizing the likelihood leads to the same objective function as minimizing the squared errors in linear regression.

What are the key assumptions made about noise in probabilistic modeling?

The primary assumptions are that the noise (epsilon) has a zero expected value (unbiased) and is independent and identically distributed (IID). These assumptions simplify analysis and often lead to using Gaussian distributions for the errors.

Why is the Gaussian distribution commonly used in probabilistic models?

Gaussian distributions have convenient mathematical properties, are characterized by mean and variance, and arise naturally from the IID Gaussian noise assumption. They serve as a good, versatile first approximation for modeling errors.

How is logistic regression derived from maximum likelihood?

By modeling the discrete 0/1 outcomes with a Bernoulli distribution and applying maximum likelihood, then taking the log, we arrive at a cost function that can be optimized using gradient descent, leading to the logistic regression model.

Why can't we simply use least squares for classification tasks?

Least squares can produce predictions outside the desired 0-1 range and is overly sensitive to outliers in classification. It tries to find a center of mass rather than a clear decision boundary, which is inappropriate for discrete labels.

What is a 'link function' in classification models?

A link function, like the sigmoid in logistic regression, transforms a linear hypothesis into a non-linear output. It's typically smooth and monotone, mapping the linear output to a probability between 0 and 1.

What is the difference between gradient descent and Newton's method for optimization?

Gradient descent takes small steps in the direction of the negative gradient, while Newton's method uses second-order information (the Hessian) to directly calculate a step that aims for the minimum. Newton's method converges faster per step but is computationally more expensive.

Why is Stochastic Gradient Descent (SGD) preferred over Newton's method in large-scale machine learning?

SGD is computationally much cheaper per step, especially with massive datasets (large N and D). Newton's method, while efficient in terms of steps, becomes prohibitively expensive due to its reliance on matrix inversions and Hessian calculations.

What is the role of 'logits' in machine learning?

Logits are the raw, untransformed output of the linear part of a model before it's passed through a link function like the sigmoid. They can sometimes be large positive or negative numbers.

Key Moments

Stanford CS229 Machine Learning | Spring 2026 | Lecture 3: Weighted Least Squares

Stanford Online

Education5 min read63 min video

Jul 29, 2026|1,752 views|28

Stanford Online Stanford Machine Learning AI Artificial Intelligence

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Least squares regression can be re-interpreted probabilistically, leading to logistic regression for classification and showcasing the power of maximum likelihood estimation.

Key Insights

The lecture re-frames linear regression as a probabilistic model by assuming errors (epsilon_i) are zero-mean, independent, and identically distributed (IID), which leads to a Gaussian distribution for the errors.

Least squares loss is equivalent to maximizing the likelihood of the data under a Gaussian noise model, justifying the least squares approach from a probabilistic perspective.

Classification (predicting discrete outcomes like 0 or 1) requires a different approach than regression; directly applying least squares can lead to predictions outside the {0, 1} range and overly sensitive decision boundaries.

Logistic regression uses a sigmoid (logistic) link function to map linear model outputs to probabilities between 0 and 1, suitable for binary classification.

The maximum likelihood principle, applied to logistic regression, leads to a convex optimization problem that can be solved using gradient descent or stochastic gradient descent.

Newton's method, a second-order optimization technique, offers fast convergence but is computationally expensive for large-scale machine learning problems compared to SGD.

From least squares to probabilistic modeling

The lecture begins by revisiting linear regression, specifically the least squares method, and proposes to justify it through a probabilistic lens. The core idea is to interpret the observed labels (y_i) as being generated by a true underlying model (theta * x_i) plus some noise (epsilon_i). By assuming this noise follows a zero-mean, independent, and identically distributed (IID) Gaussian distribution, the least squares objective function emerges naturally as the negative log-likelihood. This probabilistic formulation is crucial because it provides a principled way to extend the model to new settings, such as classification, and highlights the foundational role of maximum likelihood estimation (MLE) in machine learning.

Assumptions about noise in probabilistic regression

To derive the Gaussian noise model, specific assumptions are made about the error term (epsilon_i). First, the expected value of epsilon_i is assumed to be zero, meaning the noise is unbiased and doesn't systematically push predictions too high or too low. Second, the errors are assumed to be independent and identically distributed (IID). This IID assumption is a strong simplification that enables mathematical tractability, allowing for computations to be factored. While these assumptions might not perfectly reflect real-world data, they form a robust foundation for many machine learning models, emphasizing that models are abstractions and their utility, not absolute truth, is key.

The Gaussian distribution and its properties

The assumptions of zero mean and IID errors lead directly to the Gaussian (normal) distribution. The probability density function of a Gaussian is characterized by its mean (mu) and variance (sigma^2). The lecture explains the standard notation for a Gaussian distribution and details its functional form, highlighting that the 'action' occurs within the exponential term. This term involves the squared distance from the mean, scaled by the variance. The variance dictates the 'peakiness' or 'flatness' of the distribution: a small variance results in a sharply peaked distribution, while a large variance leads to a flatter one. This form is desirable because it's mathematically convenient, especially for optimization, and forms the basis for many statistical analyses.

Why least squares is a special case of maximum likelihood

By assuming that the errors in a linear regression model are Gaussian and IID, the objective of minimizing the sum of squared errors (least squares) is shown to be equivalent to maximizing the likelihood of observing the data given the model parameters. The process involves defining the likelihood function as the product of individual data point probabilities, converting this to a log-likelihood for numerical stability and easier computation, and then finding the parameters that maximize this log-likelihood. When applied to a linear model with Gaussian noise, this maximization procedure precisely yields the least squares solution, demonstrating that least squares is not just an ad-hoc method but a principled outcome of probabilistic modeling under specific assumptions.

Transitioning to classification: limitations of least squares

The lecture then shifts to classification, a task involving discrete outcomes (e.g., 0 or 1). A crucial question arises: why can't we simply apply least squares regression to classification problems? The answer lies in the nature of the labels. If labels are binary (0 or 1), least squares can produce predictions outside this range, making direct interpretation as probabilities impossible. Furthermore, outliers or extreme data points can disproportionately influence the regression line, leading to suboptimal decision boundaries. For example, a single point far from the main clusters can significantly 'pull' the regression line, creating a less effective separator. Thus, a dedicated approach is needed for classification.

Introducing logistic regression for binary classification

To address the limitations of least squares in classification, logistic regression is introduced. It still uses a linear model (theta * x) but passes its output through a non-linear 'link function', typically the sigmoid (or logistic) function. This sigmoid function maps any real-valued input to an output between 0 and 1, which can be interpreted as the probability of belonging to the positive class (e.g., P(y=1|x)). The formula for logistic regression is h(x) = sigmoid(theta * x). This probabilistic interpretation is powerful: the model predicts the probability of an outcome, not just a discrete label or a continuous value outside the desired range. The probability of the negative class is then simply 1 - h(x).

Maximum likelihood for logistic regression

The maximum likelihood principle is then applied to logistic regression. The likelihood function is constructed based on the binary nature of the labels (y_i being either 0 or 1), where the probability of observing y_i is p(y_i | x_i, theta) = h(x_i)^y_i * (1 - h(x_i))^(1-y_i). Taking the logarithm of this likelihood results in a sum that is more amenable to optimization. The goal is to find the parameters (theta) that maximize this log-likelihood. This optimization problem for logistic regression is convex, meaning it has a single global optimum, which can be efficiently found using optimization algorithms like gradient ascent (since we are maximizing).

Optimization methods: SGD vs. Newton's method

The lecture briefly touches upon optimization algorithms. While Stochastic Gradient Descent (SGD) is presented as the workhorse for large-scale machine learning due to its computational efficiency per step (though it may require many steps), Newton's method is also introduced as a second-order optimization technique. Newton's method uses the Hessian (second derivative) to find the minimum/maximum and converges very quickly when it does converge. However, its per-step computational cost (involving matrix inversion) is very high, making it impractical for the massive datasets and high-dimensional feature spaces common in modern machine learning. Variants of these methods are discussed, highlighting the trade-offs between computational cost per step and the number of steps required for convergence, with SGD/mini-batch SGD typically being preferred for large-scale problems.

Mentioned in This Episode

●Software & Apps

Common Questions

Regression deals with continuous outcomes (like prices), while classification handles discrete outcomes (like cat vs. dog). Classification requires different modeling approaches due to the nature of its discrete label space.

Topics

AI & Machine Learning Technology & Innovation Science & Mathematics Linear Regression Optimization Algorithms Probabilistic Modeling Logistic Regression Maximum Likelihood Estimation Gaussian Distributions

Mentioned in this video

Software & Apps

stochastic gradient descent

A workhorse optimization method used throughout the course, applicable to additive cost functions derived from maximum likelihood.

ChatGPT

Mentioned as an example where logistic regression (or softmax) forms the last layer of the model.

Newton's method

An optimization algorithm briefly introduced as historically important and interesting, contrasting with gradient descent for its speed but computational cost.

SGD

A foundational optimization method discussed previously, used here in the context of solving for model parameters.

L-BFGS

A variant of Newton's method (limited memory second-order method) used in classical statistics for optimization, especially with fewer features.

Adagrad

An algorithm inspired by estimating curvature information, related to second-order methods and used in machine learning optimization.

Adam

A popular optimizer in machine learning, described as a combination of ideas from second-order methods and momentum.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free