What is the difference between supervised and unsupervised pre-training for embeddings?

Supervised pre-training uses labeled data (e.g., ImageNet) to train a model to predict labels, with representations taken from intermediate layers. Unsupervised pre-training, like contrastive learning, learns representations without labels by optimizing similarity between augmented views of the same data.

What is contrastive learning and how does it work for images?

Contrastive learning learns representations without labels. It works by taking two augmented versions of an image and training the model to make their embeddings close, while also ensuring embeddings of different images are far apart to prevent trivial solutions.

What is the SimCLR loss function and what does it aim to achieve?

The SimCLR loss function encourages embeddings of augmented views of the same image (positive pairs) to be close, while pushing embeddings of different images (negative pairs) further apart. It's essentially a multi-class classification problem where the model identifies which embedding belongs to its 'pair'.

Are there more negative or positive pairs in contrastive learning?

Typically, there are significantly more negative pairs (augmentations of different images) than positive pairs (augmentations of the same image). This is considered reasonable as distinguishing vastly different images is easier, but the strategy is to use 'hard negatives' to make the learning more challenging and robust.

What are 'hard negatives' in contrastive learning?

Hard negatives are pairs of data that are not identical but appear similar, making them difficult to distinguish. For example, an image of a child playing soccer and text about the FIFA World Cup. Using these makes the loss function more demanding and encourages stronger representations.

How is semantic search implemented using embeddings?

To perform semantic search, you first compute embeddings for all items in your collection (e.g., documents, images). Then, you embed the query and find the items whose embeddings have the highest similarity (e.g., maximum inner product) to the query embedding.

What is retrieval augmented generation (RAG)?

RAG is a technique for large language models that supplements their existing knowledge with external, proprietary information at inference time. It involves retrieving relevant documents and providing them as context to the LLM along with the query to generate a more informed answer.

What are the benefits of using RAG over fine-tuning LLMs?

RAG is modular, cost-effective, and simplifies data governance. It allows for easy addition or deletion of information without retraining the LLM, whereas fine-tuning can be expensive and complex, especially for managing and unlearning information.

Are there alternatives to semantic search for retrieval in LLMs?

Yes, some models like those from Anthropic might use regular expression matching, especially for structured data like code, where regex can effectively capture patterns. However, semantic retrieval remains useful for more unstructured or nuanced content.

Key Moments

Stanford CS229 Machine Learning | Spring 2026 | Lecture 13: LLMs, Next-Word Prediction Loss

Stanford Online

Education6 min read61 min video

Jul 31, 2026|283 views|12|1

Stanford Stanford Online Machine Learning Artificial Intelligence AI

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Contrastive learning for embeddings maps similar items closely and dissimilar items far apart, avoiding the need for expensive labeled data, but may struggle with nuanced similarities.

Key Insights

Representation learning maps raw inputs (images, text, audio) into a vector space (embeddings) where similar items are close and dissimilar items are far apart.

Supervised pre-training for embeddings involves training a network on a large labeled dataset (e.g., ImageNet with 1000 labels) and using the penultimate layer's output as representations.

Contrastive learning aims to learn embeddings without labels by augmenting an image to create two versions, encouraging their embeddings to be close while pushing embeddings of different images apart.

The SimCLR loss function can be interpreted as a multiclass classification problem where the model distinguishes an augmented image from augmentations of other images.

Semantic search finds similar items by computing embeddings for a collection and then finding the nearest neighbor to a query embedding, often accelerated by vector databases.

Retrieval Augmented Generation (RAG) allows large language models to access proprietary or additional information at test time by retrieving relevant documents and including them in the model's context, without costly fine-tuning.

The goal of representation learning and embeddings

Representation learning focuses on creating a mapping function, parameterized by theta, that transforms raw input data (like images, text, or audio) into a vector space. These vectors are commonly referred to as embeddings. The core principle is that similar inputs should be mapped to similar embeddings, meaning their vectors are close in the Euclidean space. Conversely, dissimilar inputs should be mapped to embeddings that are far apart. This allows for applications like similarity search, where given an embedding of a cat image, one can find other similar cat images by searching for the nearest neighbors in the embedding space. While embeddings are sometimes called representations or features, 'embeddings' is the most prevalent term today. This concept is fundamental for various data types, though images often serve as the canonical example for explanation.

Supervised pre-training for embeddings

One method to obtain embeddings is supervised pre-training. This involves using a network trained on a large-scale supervised classification task, such as ImageNet with its 1000 labels. The network is trained to predict these labels, and the representations are extracted from the penultimate layer (the layer just before the final classification output). The assumption is that to accurately classify a diverse set of labels, the network must learn features that capture meaningful similarities and differences in the data. However, this approach has limitations: it requires a vast amount of labeled data, which is expensive and time-consuming to acquire. The quality of the learned representations is also tied to the diversity of the labels; limited labels might only capture narrow patterns, making the embeddings less useful for other downstream tasks.

Contrastive learning: learning embeddings without labels

To overcome the limitations of supervised pre-training, unsupervised approaches like contrastive learning have emerged. The fundamental idea is to learn representations without needing any explicit labels. The process begins with an input image, which is then augmented (e.g., through random cropping, flipping, blurring, or color transformations) to create two or more modified versions. For example, two random crops of the same image can serve as an augmentation pair. A neural network (parameterized by theta) is applied to these augmented images to generate embeddings. The primary objective is to minimize the distance between the embeddings of these augmented versions of the *same* image, encouraging them to be close in the embedding space. This objective is crucial for ensuring that semantically similar inputs are mapped to nearby points.

Balancing proximity and distance in contrastive learning

While bringing augmentations of the same image closer is important, it's insufficient on its own. If this were the only objective, the model would simply map all inputs to the same point, rendering the embeddings useless. To counteract this, a balancing objective is needed: pushing embeddings of *different* images further apart. This is achieved by taking an embedding from a different original image (and its augmentations) and encouraging it to be distant from the embeddings of the first image. The challenge here is that random sampling of images means that occasionally, two different images might actually be similar (e.g., two different cats). In such cases, the objective to push them apart is counterproductive. However, the lecture notes that in most scenarios, randomly sampled images are indeed dissimilar, and the benefit of pushing genuinely different items apart outweighs the minor damage caused by pushing slightly similar but distinct items apart.

The SimCLR loss function and its interpretation

The SimCLR framework formalizes this contrastive learning objective. It involves sampling a batch of images and creating two augmented versions for each. These are then embedded using a neural network. A similarity matrix is constructed where entries represent the similarity (typically cosine similarity or normalized inner product) between embeddings of augmented images. The goal is to have large values on the diagonal (representing pairs from the same original image) and small values in the off-diagonal entries (representing pairs from different original images). The loss function used is essentially a cross-entropy loss for a multi-class classification problem. For each augmented image (query), the model tries to classify which of the other augmented images in the batch is its positive pair (from the same original image), distinguishing it from many negative pairs (from different original images). This formulation makes the loss demanding, forcing the model to learn robust representations.

Semantic search and its implementation

Embeddings are highly effective for semantic search. The process involves pre-computing the embeddings for a large collection of items (documents, images, etc.) and storing them. When a query arrives, its embedding is computed. The task then becomes finding the item in the collection whose embedding is closest to the query embedding. This is a nearest neighbor search problem. A brute-force approach involves calculating the similarity between the query embedding and every item embedding in the collection and selecting the top results. For efficiency, specialized vector databases and k-nearest neighbor algorithms are employed to speed up this search process, making it practical for large-scale applications.

Retrieval Augmented Generation (RAG) for LLMs

Large language models (LLMs) often lack access to proprietary or domain-specific information. Fine-tuning LLMs on such data can be prohibitively expensive. Retrieval Augmented Generation (RAG) offers a solution by integrating retrieval with generation. When a query is made, the RAG system first retrieves relevant documents from a private corpus using semantic search. A subset of these retrieved documents is then provided as context to the LLM along with the original query. The LLM generates a response based on both its internal knowledge and the provided context. This approach is modular, cost-effective, and allows for easier data management, such as controlling access to sensitive information or enabling information deletion (un-forgetting).

Advanced retrieval techniques and future directions

While semantic search is a powerful retrieval method for RAG, other techniques are also explored. For instance, some systems might use LLMs to generate complex regular expressions to search for specific patterns, which can be particularly effective for structured data like code. The choice between semantic retrieval, pattern-based retrieval, or a hybrid approach often depends on the specific application and data characteristics. The field is continuously evolving, with ongoing research into optimizing retrieval strategies to enhance the performance and applicability of LLMs in various real-world scenarios. The development of 'hard negative mining' also plays a crucial role, where the system actively seeks out challenging negative examples that are subtly similar to positive examples, thus making the training process more robust.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

Common Questions

Embeddings are vector representations of data (like images, text, or audio) in an 'embedding space'. The goal is for similar inputs to be mapped to similar vectors, and dissimilar inputs to be mapped to distant vectors, enabling tasks like similarity search.

Topics

AI & Machine Learning Technology & Innovation Science & Mathematics Large Language Models Retrieval-augmented Generation Representation Learning Unsupervised Learning Semantic Search Contrastive Learning

Mentioned in this video

Software & Apps

ImageNet

A large-scale dataset used for training image classification models, often featuring around a thousand labels, which can be used for supervised pre-training to learn representations.

SimCLR

A contrastive learning framework for unsupervised representation learning that aims to learn embeddings by bringing augmentations of the same image closer while pushing apart embeddings of different images.

Companies

Anthropic

A frontier AI research company mentioned as using regular expression matching for retrieval in large language models, distinct from pure semantic search.

Google

Mentioned as a search system that likely uses a fusion of methods, including semantic search, though its exact implementation is not public.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free