What is the general framework for supervised learning?

The general framework involves a dataset of input-output pairs, a model parameterized by theta (h(theta)), a loss function to measure prediction error, and an optimizer to minimize the loss function by adjusting theta. This applies to both regression and classification tasks.

How does cross-entropy loss work in multiclass classification?

Cross-entropy loss is used for multiclass classification by defining a model that outputs probabilities for each class using a softmax function. The loss is then calculated as the negative log-likelihood of the true label given the model's predicted probabilities, measuring the difference between the predicted and true distributions.

Why is stochastic gradient descent (SGD) used instead of standard gradient descent?

SGD is used because computing the full gradient over massive datasets is computationally prohibitive. SGD uses a single sample or a small batch to approximate the gradient, making training feasible. While noisier, it is more efficient and can sometimes perform better on non-convex functions.

What is the main idea behind neural networks?

Neural networks are nonlinear models composed of layers. Each layer typically involves a linear transformation (matrix multiplication and bias addition) followed by a nonlinear activation function (like ReLU). These layers are stacked to create complex, hierarchical representations of data.

What is a Rectified Linear Unit (ReLU) and why is it important?

ReLU is an activation function defined as max(t, 0). It introduces nonlinearity into neural networks by outputting the input if it's positive, and zero otherwise. This simple yet effective function is a fundamental building block in deep learning.

How are neural networks structured using matrices and vectors?

Neural networks are structured using matrix multiplications for linear transformations and vector biases. Multiple layers are formed by composing these operations, where the output dimension of one layer becomes the input dimension for the next, ensuring compatibility for matrix operations.

What are common activation functions used in neural networks?

Besides ReLU, other activation functions include sigmoid (which was used early on), tanh, leaky ReLU, and others. These functions introduce nonlinearity, allowing networks to learn complex patterns. The choice often depends on empirical performance and specific architectural needs.

What is a residual connection in neural networks?

A residual connection allows the input of a block to be added back to its output. This helps in training very deep networks by ensuring that layers can easily learn identity mappings if needed, preventing performance degradation as depth increases.

What is Layer Normalization and why is it used?

Layer Normalization is a technique that normalizes the inputs to a layer across their features. It helps stabilize training, allows for higher learning rates, and makes the model less sensitive to the scale of its parameters and data, which can prevent issues like exploding outputs.

What is the basic concept behind Convolutional Networks?

Convolutional networks use matrices with specific structures (like filters) that are applied across the input data. This involves parameter sharing and local connectivity, which is particularly effective for processing grid-like data such as images, by detecting spatial hierarchies of features.

Key Moments

Stanford CS229 Machine Learning | Spring 2026 | Lecture 7: Neural Networks 1 (Architecture)

Stanford Online

Education6 min read81 min video

Jul 30, 2026|3,015 views|123|2

Stanford Stanford Online Machine Learning Artificial Intelligence AI

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Neural networks can learn complex nonlinear relationships, but training them requires backpropagation to compute gradients, and stochastic gradient descent is often used due to its efficiency on large datasets.

Key Insights

The core difference between linear and nonlinear models lies in their parameterization; models like theta * x^2 are considered linear from a learning perspective if the non-linearity can be transformed into a linear form, whereas models with non-linearity in parameters (e.g., theta^2 * x) are truly nonlinear.

For regression, a common loss function is the mean squared error: (y - h(x))^2, which simplifies to mean squared error if the model assumes a Gaussian distribution for the error term.

In multiclass classification, the cross-entropy loss is used, calculated as the negative log-likelihood of the true label given the model's predicted probabilities (often derived from a softmax function applied to logits).

Stochastic Gradient Descent (SGD) updates parameters using the gradient of a single data point's loss, which is computationally cheaper and, on average, points in the same direction as the gradient of the average loss, though with more noise.

The Rectified Linear Unit (ReLU) activation function, defined as max(t, 0), is a fundamental building block in modern neural networks, introducing non-linearity by zeroing out negative inputs.

Residual connections, where the input is added back after passing through one or more layers (e.g., z + f(z)), help stabilize training by allowing models to learn the difference (residual) between the input and output, especially in deep networks.

Distinguishing linear from nonlinear models

The lecture begins by clarifying the distinction between linear and nonlinear models in machine learning. A linear model is characterized by a linear combination of parameters, such as \( \theta_1 x_1 + \theta_2 x_2 \). While models like \( \theta_1 x_1^2 + \theta_2 x_2^2 \) involve non-linear transformations of the input features (e.g., \( x_1^2 \) can be replaced by a new feature \( z_1 \)), they can still be treated as linear models from a learning perspective because the relationship with the parameters \( \theta \) remains linear. True nonlinearity, as discussed in the context of neural networks, arises when the model's parameters themselves are involved in a nonlinear function, such as \( \theta_1^2 x_1 \). This fundamental difference dictates the complexity of learning algorithms required.

Supervised learning framework: regression and classification

The general framework for supervised learning is introduced, starting with regression. For a regression problem, the goal is to predict a continuous real-valued output \( y \) from an input \( x \). The model, denoted as \( h_\theta(x) \), maps the input \( x \) to an output. A common loss function for regression is the mean squared error, \( (y - h_\theta(x))^2 \), averaged over all data points. This is motivated by assuming the output \( y \) is generated by the model \( h_\theta(x) \) plus Gaussian noise, where the negative log-likelihood under this assumption leads to the mean squared error. For multiclass classification, where \( y \) can take one of \( k \) discrete labels, the model outputs \( k \) logits, which are then passed through a softmax function to produce probabilities for each class. The objective is typically to minimize the negative log-likelihood, which translates to the cross-entropy loss between the true distribution of labels and the model's predicted distribution.

Optimization: Gradient Descent and its stochastic variant

To train these models, optimization techniques are employed to minimize the overall loss function. Gradient Descent is a fundamental algorithm where parameters \( \theta \) are updated iteratively by moving in the opposite direction of the gradient of the loss function: \( \theta \leftarrow \theta - \epsilon \nabla J(\theta) \). However, computing the gradient over the entire dataset can be computationally prohibitive for large datasets. Stochastic Gradient Descent (SGD) offers an efficient alternative by computing the gradient using only a single, randomly sampled data point at each step. While this introduces noise into the update direction, the expected gradient of a single sample is equal to the gradient of the average loss. This makes SGD a practical choice for large-scale problems, though mini-batch SGD, which uses a small batch of samples, is often used to balance computational efficiency and gradient accuracy, and to better utilize hardware like GPUs.

The Rectified Linear Unit (ReLU) and neuron activation

Neural networks introduce nonlinearity through activation functions. The Rectified Linear Unit (ReLU), defined as \( \text{ReLU}(t) = \max(t, 0) \), is a key primitive. It applies a linear transformation \( Wx + b \) to the input and then passes the result through the ReLU function. This effectively 'rectifies' the output, setting all negative values to zero. The term 'activation function' is inspired by biological neurons, where a neuron fires only when its input stimulus exceeds a certain threshold. While early networks used sigmoid functions, ReLU became popular due to its computational simplicity and effectiveness in mitigating issues like the vanishing gradient problem.

Building neural networks: layers and parameterization

A single layer of a neural network typically consists of a linear transformation (matrix multiplication with weights \( W \) and addition of bias \( b \)) followed by an activation function. For an input vector \( x \) of dimension \( d \) and an output vector \( a \) of dimension \( m \), the weight matrix \( W \) would have dimensions \( m \times d \), and the bias vector \( b \) would have dimension \( m \). The total number of parameters for this layer is \( md + m \). Deeper networks are formed by stacking these layers, where the output of one layer serves as the input to the next. The dimensionality of the intermediate layers can be chosen, often decreasing from the input dimension to manage complexity. This composition of linear transformations and nonlinear activations forms the basis of multi-layer perceptrons (MLPs).

Residual connections for deeper architectures

To train very deep neural networks effectively, residual connections were introduced. Instead of simply passing the output of a layer \( z \) through another transformation \( f \), a residual block computes \( z + f(z) \). This allows the network to easily learn identity mappings (if \( f(z) \) is close to zero) and learn the residual or difference between the input and the desired output. This technique has been crucial in enabling the training of networks with hundreds or even thousands of layers, improving performance and stability by mitigating issues related to vanishing gradients and making optimization easier. The original motivation was that if the current intermediate representation \( z \) is already close to the target \( y \), it's simpler to learn the difference \( y - z \) than to learn \( y \) from scratch.

Normalization techniques: Layer Normalization

Layer Normalization (LayerNorm) is a component used to stabilize training by normalizing the inputs to a layer. It computes the mean \( \mu \) and standard deviation \( \sigma \) of the inputs across the feature dimension and then normalizes the inputs to have zero mean and unit variance. Additionally, it introduces learnable scale (gamma) and shift (beta) parameters, allowing the network to adjust the normalized values. This normalization makes the network less sensitive to the scale of inputs and initial parameter values, aiding convergence. A simpler variant, RMSNorm, omits the mean subtraction, relying only on normalization by the standard deviation, and also incorporates learnable parameters.

Mentioned in This Episode

●Software & Apps

●Concepts

Common Questions

Linear models have a functional form that is linear in the parameters, meaning transformations of inputs can still result in a linear model. Nonlinear models, however, have parameters that appear in nonlinear functions (like squares or other non-linear operations), preventing simple transformations to a linear form and requiring different learning techniques.

Topics

Learning & Education AI & Machine Learning Technology & Innovation Science & Mathematics Model Architecture Neural Networks Deep Learning Gradient Descent Supervised Learning Activation Functions

Mentioned in this video

Software & Apps

PyTorch

Mentioned as a framework where a cross-entropy loss module can be found.

Transformers

Mentioned as complex nonlinear functions used in language cases.

Concepts

RM Norm

A simpler normalization technique that omits subtracting the mean, focusing only on division by standard deviation, with tunable parameters.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free