How are diffusion models trained?

Diffusion models are trained by minimizing a loss function that aims to predict the noise added during the forward diffusion process. The model takes a noisy input XT and learns to predict the noise epsilon that was originally added.

What is the core idea behind foundation models?

Foundation models are trained on massive amounts of unlabeled data using general-purpose objectives. This pre-training creates a strong base that can then be adapted to a wide range of downstream tasks with less task-specific data.

What is representation learning?

Representation learning involves training a model to output a fixed-size vector (a representation or embedding) from input data. This representation is then used for downstream tasks, often with a simple linear model on top.

What is linear probing in the context of representation learning?

Linear probing involves training a simple linear classifier or regressor on top of a fixed, pre-trained representation. The pre-trained model's parameters are frozen, and only the weights of the linear layer are optimized.

What is the difference between linear probing and fine-tuning?

Linear probing freezes the pre-trained model's parameters and only trains a linear layer. Fine-tuning updates both the pre-trained model's parameters and the parameters of the new layer, adapting the entire model to the downstream task.

What is LoRA (Low-Rank Adaptation)?

LoRA is a technique that significantly reduces the number of trainable parameters during fine-tuning. It freezes the pre-trained model's weights and injects small, trainable low-rank matrices into specific layers.

What are the main benefits of using LoRA?

LoRA drastically reduces memory requirements and enables efficient serving of multiple fine-tuned models by allowing users to share the base model weights and only load small, user-specific low-rank adapters.

How does LPFT (Linear Probing Fine-Tuning) work?

LPFT first performs linear probing to optimize a linear head, then uses the resulting initialized weights to fine-tune both the linear head and the pre-trained model parameters.

Why is initialization important for fine-tuning?

Initialization with pre-trained weights is crucial because the loss function for fine-tuning often has multiple global minima. A good initialization guides the optimization process towards a minimum that yields better test performance.

Key Moments

Stanford CS229 Machine Learning | Spring 2026 | Lecture 12: Representation Learning

Stanford Online

Education5 min read76 min video

Jul 31, 2026|923 views|19

Stanford Online Stanford Machine Learning Artificial Intelligence AI

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Stanford's CS229 lecture on diffusion models and foundation models covers advanced AI techniques and introduces a new paradigm for machine learning using massive datasets.

Key Insights

The training loss for diffusion models can be simplified to minimizing the difference between the added noise and the predicted noise, making the process more robust.

Foundation models represent an emergent paradigm in machine learning, characterized by pre-training on massive, unlabelled datasets and subsequent adaptation to a wide range of tasks.

Linear probing, a technique in representation learning, involves training a linear classifier on top of a fixed, pre-trained feature extractor, demonstrating its effectiveness even with limited downstream labels.

Fine-tuning updates both the pre-trained parameters and the downstream task parameters, with pre-trained initialization significantly impacting the final model performance.

LoRA (Low-Rank Adaptation) offers a memory-efficient method for fine-tuning large models by updating only a low-rank decomposition of the weight matrices, allowing multiple users to share a base model and significantly reducing serving costs.

Understanding diffusion model training and loss simplification

The lecture begins by revisiting diffusion models, explaining their forward noising and reverse denoising processes. The core of diffusion model training lies in maximizing the log-likelihood of the data, which is typically optimized using a lower bound (ELBO). A key derivation shows that for diffusion models, the loss function simplifies significantly. Specifically, the KL divergence terms in the ELBO, when comparing Gaussian distributions, primarily depend on the difference between the predicted mean and the true mean. The lecture highlights that the loss can be reduced to a mean squared error between the actual noise added and the noise predicted by the neural network. This simplification, often dropping coefficients in practice, turns the complex generative task into a more manageable noise prediction problem. The process involves sampling an initial data point, a time step, adding noise to create a noisy version, and then training a neural network to predict this added noise. This noise prediction formulation is central to the training algorithm.

The emergence of foundation models and their two-phase approach

The discussion shifts to foundation models, a paradigm shift in machine learning observed around 2020, notably influenced by models like GPT-3. This new paradigm is characterized by two main phases: pre-training and adaptation. The pre-training phase involves using massive amounts of unlabelled data, often scraped from the internet, without explicit task-specific labels. This contrasts with previous approaches that focused on single tasks or limited multitask learning. The scale of data is magnitudes larger than before, and the diversity of data is also immense. The goal of pre-training is to build a robust, general-purpose model with a strong 'foundation.' This foundation can then be adapted to a vast, almost unlimited number of downstream tasks, moving beyond the fixed set of tasks common in earlier machine learning research.

Representation learning and linear probing for adaptation

Representation learning, a key aspect of adapting foundation models, focuses on training a model (denoted as 'f' with parameters 'theta') to map input data 'X' into a lower-dimensional vector space. These output vectors are called representations, features, or embeddings. The goal is to learn representations that capture essential semantic information. Once these representations are learned (through pre-training), a downstream task can be solved using a simple linear model (a linear head 'W'). This technique, called linear probing, involves freezing the pre-trained representation model and only training the linear classifier on top of these learned features. It's a valuable method for downstream tasks where labeled data is scarce, as demonstrated in domains like medical imaging, where a general visual representation model is adapted to various specific imaging tasks using only a small amount of labeled data for each task.

Fine-tuning as a parameter optimization strategy

Fine-tuning offers a more extensive adaptation method compared to linear probing. In fine-tuning, both the parameters of the pre-trained model ('theta') and the parameters of the downstream task's linear head ('W') are updated. The process starts by initializing 'theta' with the pre-trained parameters and 'W' often randomly. The model is then trained on the downstream task's labeled data by minimizing a loss function that updates both sets of parameters. While this approach can yield better results than linear probing, it requires more computational resources and is more prone to overfitting due to the larger number of parameters being updated. The effectiveness of fine-tuning highlights the importance of initialization; starting with a good pre-trained model significantly improves the chances of finding a better global minimum of the loss function for the downstream task.

LoRA: efficient adaptation through low-rank updates

LoRA (Low-Rank Adaptation) is presented as a highly efficient adaptation technique, particularly relevant for very large models where full fine-tuning is computationally prohibitive. Instead of updating all parameters of the pre-trained model ('theta'), LoRA introduces a low-rank decomposition update. For each weight matrix 'W' in the pre-trained model, LoRA adds a low-rank matrix 'A' and 'B' such that the updated weight becomes 'W + AB'. The matrices 'A' and 'B' have significantly fewer parameters than 'W', thereby restricting the degrees of freedom in the update. This approach saves memory and computation during training and, crucially, during inference. A key benefit is the ability to serve multiple users with a single base model ('W0') by swapping out their personalized low-rank adapters ('A', 'B'), drastically reducing serving costs and enabling faster task switching for users.

The layered nature of machine learning optimization

The lecture touches upon the layered nature of machine learning algorithms, differentiating between innovations in the loss function, architecture, and optimizers. While much of the discussion focuses on the loss and architecture, the underlying optimization algorithms (like SGD or Adam) are also critical components. Changes in optimizers can have universal applicability across different models and tasks. The core premise is that once a loss function is defined, standard auto-differentiation and optimization techniques can be applied to train the model, whether it's a diffusion model or a foundation model.

Mentioned in This Episode

●Software & Apps

●Companies

●Concepts

●People Referenced

Common Questions

Diffusion models utilize a fixed noising function for the forward process and learn a denoising function for the reverse process. This decomposition simplifies optimization and learning compared to models that parameterize both directions.

Topics

AI & Machine Learning Technology & Innovation Diffusion Models Foundation Models Representation Learning Linear Probing Parameter Efficiency

Mentioned in this video

Software & Apps

GPT-3

Mentioned as a breakthrough model that enabled few-shot and zero-shot learning, significantly impacting the adaptation phase in machine learning.

GPT-4

Mentioned as an example of a large model (W0) that can be shared across multiple users when using techniques like LoRA.

Gemini

Mentioned alongside GPT in the context of offering fine-tuning services that likely utilize LoRA.

Companies

OpenAI

The company behind GPT-3, whose playground was used to demonstrate the excitement around early large language models.

People

Percy Liang

Led the team that studied the new paradigm of foundation models and co-authored the survey paper introducing the term.

Concepts

LoRA

A technique for efficient model fine-tuning that restricts updates to low-rank matrices, significantly reducing memory and computational requirements, and enabling easier serving of multiple users.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free