Why is subword tokenization preferred over word or character tokenization?

Subword tokenization offers a balance. Character tokenization can lead to very long sequences, while word tokenization struggles with rare or novel words. Subword tokenization breaks down words into smaller meaningful parts, allowing the model to leverage known subwords to understand new words more efficiently and handle morphological variations.

How does an autoregressive language model generate text?

Autoregressive models generate text token by token, where each new token depends on the previously generated tokens. This sequential process starts with an initial prompt or a beginning-of-sentence token and iteratively predicts the next most likely token until a stopping condition is met.

What is the role of embeddings in a transformer model?

Embeddings convert discrete token IDs into dense numerical vectors in a high-dimensional space. These embeddings capture semantic relationships between tokens and are the initial input to the transformer layers, allowing the model to process linguistic information numerically.

How does temperature sampling affect text generation in LLMs?

Temperature sampling controls the randomness of token selection during generation. A low temperature makes the output more deterministic and focused on the most likely tokens, while a higher temperature increases randomness, leading to more diverse and potentially creative, but sometimes less coherent, outputs.

What is the core function of the attention mechanism in transformers?

The attention mechanism allows the model to weigh the importance of different input tokens when processing a specific token. It calculates relevance scores between a 'query' token and 'key' tokens, then uses these scores to create a weighted sum of 'value' tokens, enabling the model to focus on relevant parts of the input sequence.

Why is masking necessary in the attention mechanism for autoregressive models?

Masking prevents the model from attending to future tokens during generation. In autoregressive models, the output at time step 't' should only depend on tokens up to 't-1'. Masking ensures this by setting attention scores for future tokens to negative infinity, effectively zeroing them out after the softmax.

What is multi-head attention and why is it used?

Multi-head attention runs the attention mechanism multiple times in parallel with different learned linear projections (weights). This allows the model to jointly attend to information from different representation subspaces at different positions, capturing a wider range of relationships and improving performance.

What are residual connections in transformers, and why are they important?

Residual connections add the input of a layer (or a block of layers) to its output. This technique, common in deep networks, helps gradients flow more easily through the network, preventing the vanishing gradient problem and enabling the training of much deeper models.

Key Moments

Stanford CS229 Machine Learning | Spring 2026 | Lecture 14: Transformers, In-Context Learning

Q: What is the main computational challenge with standard transformer attention?

The primary challenge is the quadratic complexity (O(T^2)) with respect to the sequence length (T). This makes processing very long sequences computationally expensive and memory-intensive, often necessitating techniques to shorten context or improve efficiency.

Stanford Online

Education6 min read78 min video

Jul 31, 2026|678 views|20|2

Stanford Stanford Online Artificial Intelligence AI Machine Learning

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Subword tokenization is crucial for LLMs, breaking down rare words into understandable components, but it can lead to increased token counts and costs, as seen with cloud provider tokenizer updates.

Key Insights

Subword tokenization, such as Byte Pair Encoding (BPE), is preferred over word or character tokenization for LLMs to handle rare and complex words by breaking them into smaller, more frequent units.

Large language models process text by converting tokens into numerical embeddings, which are then fed into a transformer architecture that models conditional probabilities sequentially.

The core of the transformer is the attention mechanism, which allows the model to weigh the importance of different input tokens by calculating query-key inner products and using the resulting probabilities to aggregate value vectors.

Auto-regressive models generate text token by token, predicting the next token based on all previously generated tokens, a process that can be modified by temperature sampling to control output determinism.

The computational complexity of the standard transformer attention mechanism scales quadratically with sequence length (O(T^2)), posing challenges for processing long contexts, though techniques like FlashAttention aim to optimize memory usage.

Multi-head attention enhances the model's capacity by allowing different attention 'heads' to focus on different aspects of the input, with their outputs concatenated and projected to form the final representation.

The necessity of subword tokenization for LLMs

Large language models (LLMs) require numerical input, necessitating a process called tokenization to convert text into a format the model can understand. While character or word-level tokenization has drawbacks (too many tokens for characters, difficulty with rare words for words), subword tokenization is the prevailing method. This approach breaks down words into smaller, frequently occurring units, allowing the model to infer the meaning of rare or novel words by combining the meanings of their constituent subwords. For instance, a word like 'lumimications' (a hypothetical new word) could be tokenized into 'lumi', 'fications', enabling the model to leverage its understanding of 'lumi' and common English suffixes. Tokenizers, like Byte Pair Encoding (BPE), create a predefined vocabulary of these subwords. Modern open-source models like Qin 3.5 utilize vocabularies of around 250,000 tokens. The process converts text into a sequence of token IDs, with each ID corresponding to a specific subword in the vocabulary.

Modeling language distributions with auto-regressive transformers

The challenge of modeling the vast number of possible token sequences is addressed by decomposing the joint probability distribution into a product of conditional probabilities using the chain rule. This auto-regressive approach models P(X_t | X_1, ..., X_{t-1}), predicting the next token X_t given all preceding tokens. Each conditional probability is typically parameterized by a neural network, specifically a transformer. The input text is first converted into embeddings – dense vector representations – for each token. These embeddings are then processed through the transformer network, which outputs logits. These logits are then converted into a probability distribution over the entire vocabulary using a softmax function, indicating the likelihood of each token being the next in the sequence. This sequential prediction forms the basis of auto-regressive language modeling, where the model generates tokens one by one, with each new token's probability depending on the ones generated before it.

The attention mechanism: Query, Key, and Value

The transformer architecture relies heavily on the attention mechanism, which allows the model to dynamically weigh the importance of different parts of the input sequence. This mechanism operates using three types of vectors derived from the input embeddings: queries (Q), keys (K), and values (V), all generated by multiplying the input embeddings with learned weight matrices (W_q, W_k, W_v). To determine how much attention a particular token (represented by its query vector) should pay to other tokens (represented by their key vectors), the model computes the dot product between the query and each key. These dot products are then scaled and passed through a softmax function to produce attention weights – a probability distribution over the input sequence. These weights are then used to compute a weighted sum of the value vectors, producing the output representation for that token. This process enables the model to selectively focus on relevant information from the input, regardless of its position.

Masked self-attention for auto-regressive generation

For auto-regressive language generation, where the model predicts tokens sequentially without seeing future tokens, a crucial modification called 'masked self-attention' is employed. During the attention calculation, a mask is applied to prevent any token from attending to subsequent tokens in the sequence. This is achieved by adding negative infinity to the dot products of queries and keys corresponding to future positions before the softmax operation. This effectively sets the attention weights for these positions to zero, ensuring that the output at time step 't' is only influenced by tokens from time step '1' to 't'. This strict adherence to sequential dependency is fundamental for generative tasks, mirroring the process of human language production where one cannot know future words when speaking or writing.

Multi-head attention and transformer layers

To enhance the model's ability to capture diverse relationships within the data, transformers utilize multi-head attention. Instead of performing attention once, the input is processed by multiple 'heads' in parallel, each with its own set of learned query, key, and value projection matrices. Each head can potentially learn to focus on different types of information or relationships. The outputs from all attention heads are then concatenated and passed through a final linear projection layer to produce a single output vector. The transformer architecture itself is typically composed of multiple layers, each containing a multi-head attention sub-layer followed by a feed-forward network (often a simple multi-layer perceptron or MLP). Residual connections and layer normalization are often incorporated around these sub-layers to facilitate training of deep networks.

Computational complexity and optimization strategies

A significant challenge with the standard transformer architecture, particularly the self-attention mechanism, is its quadratic computational complexity with respect to the input sequence length (O(T^2)). This makes processing very long sequences computationally expensive and memory-intensive. Techniques like FlashAttention have been developed to optimize memory usage by efficiently computing attention without materializing the full T x T attention matrix. Future lectures will explore other transformer variants designed to reduce this quadratic dependency, potentially achieving linear or near-linear scaling with sequence length. However, these architectural modifications can sometimes lead to trade-offs, potentially reducing the model's expressiveness or performance.

Auto-regressive generation and temperature sampling

Once a transformer model is trained, it can be used for text generation through an auto-regressive process. Starting with an initial prompt (or a beginning-of-sentence token), the model iteratively predicts the next token by sampling from the probability distribution generated by the softmax output. This generated token is then fed back into the model as input for predicting the subsequent token. The randomness of this generation process can be controlled using a 'temperature' parameter. A low temperature sharpens the probability distribution, leading to more deterministic and focused output (closer to selecting the most likely token), while a high temperature flattens the distribution, increasing stochasticity and encouraging diversity in the generated text. Techniques like top-K sampling can also be used to limit the sampling pool to the most probable tokens.

Mentioned in This Episode

●Software & Apps

●Companies

●Concepts

Common Questions

Tokenization is the process of breaking down text into smaller units (tokens) that models can understand. It's crucial because LLMs operate on numerical data, and tokenization converts text into a sequence of numerical IDs representing these units, allowing the model to process and learn from language.

Topics

AI & Machine Learning Technology & Innovation Programming & Software Large Language Models Natural Language Processing Attention Mechanism Autoregressive Models Deep Learning Architecture

Mentioned in this video

Software & Apps

GPT-3

Mentioned as the starting point for the current paradigm of autoregressive large language models.

Claude

Mentioned in the context of updating tokenizers, potentially increasing the number of tokens for the same amount of text and leading to higher costs.

SGD

Mentioned as an optimizer used for training, alongside its extension, IDOM.

flash attention

A method proposed to reduce the memory footprint in transformer systems.

Companies

OpenAI

The company that released GPT-3, marking a significant shift in large language models.

Concepts

Byte Pair Encoding

The algorithm used for tokenization, which identifies frequent subwords to build a vocabulary.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free