Why use character-level tokenization instead of subword tokens?

Character-level tokenization keeps the implementation and vocabulary small and is educational; subword tokenizers (sentencepiece / tiktoken / BPE) trade larger vocabularies for shorter sequences and are typically used in production. See the discussion at 868 seconds.

How do you avoid feeding an entire book into the Transformer at once?

Train on random fixed-length chunks (block_size / context length) and use batches of these chunks; the model learns to predict the next token inside each chunk. This batching/block strategy is explained at 1332 seconds.

What is self-attention and how does it compute token affinities?

Self-attention computes data-dependent affinities by creating queries and keys for each position and taking scaled dot-products; masking prevents future tokens from leaking into past positions. See the attention explanation at 4031 seconds.

Why scale the dot-product (divide by sqrt(dk)) in attention?

Scaling keeps the variance of the dot-product controlled so softmax isn't pushed to extremely peaky distributions at initialization, improving training stability. This is covered at 4031 seconds (scaled dot-product).

What tricks help train deeper Transformer stacks reliably?

Use residual (skip) connections, LayerNorm (the lecture implements pre-norm), and reasonable initialization; these help gradients flow and stabilize optimization. See the residual & LayerNorm discussion at 5101 seconds.

How much compute/data is required to scale to GPT-3 style models?

Scaling to models like GPT-3 requires orders of magnitude more parameters and training tokens (GPT-3 used ~175B parameters and hundreds of billions of tokens), and extensive multi-GPU infrastructure; see the comparison at 6615 seconds.

What's the difference between pre-training and the fine-tuning used for ChatGPT?

Pre-training learns to complete text from large web-scale corpora (unsupervised). Fine-tuning (supervised + RLHF steps) aligns the model to answer questions and behave as an assistant. The two-stage overview is at 6535 seconds.

How do you generate text from a trained Transformer model?

Start with an initial context (e.g., newline / start token), run the model to get logits for the next token, convert logits to probabilities (softmax), sample (or take argmax), append and repeat. See the generate routine at 2534 seconds.

Which libraries and tools did the lecture use to implement NanoGPT?

The implementation uses PyTorch for tensors and modules, Google Colab for notebooks, and references tokenizers such as sentencepiece and tiktoken for comparison. See tool mentions at 774 and 658–703 seconds.

Let's build GPT: from scratch, in code, spelled out.

Andrej Karpathy

Science & Technology4 min read117 min video

Jan 17, 2023|6,933,611 views|152,124|3,041

deep learning neural network language model pytorch gpt chatgpt openai generatively pretrained transformer attention is all you need self-attention

Save to Pod

Key Moments

TL;DR

From scratch to GPT: building a Transformer, training on Shakespeare, and scaling.

Key Insights

GPT is a decoder‑only Transformer (Generative Pre‑trained Transformer) that learns to predict the next token in a sequence by modeling word (or character) dependencies with attention.

A tiny but powerful hands‑on dataset (tiny Shakespeare) demonstrates core training steps: character‑level tokenization, a 1 MB dataset, 65‑token vocabulary, and 90/10 train/validation split.

Self‑attention with Q, K, V, and a masked triangular attention matrix enables autoregressive generation; scaling by the head size stabilizes softmax and improves optimization.

Progression from a simple embedding‑based model to a full Transformer with multi‑head attention, residual connections, layer normalization, and dropout dramatically boosts performance and trainability.

Scaling the model (longer context, more layers, more heads, larger embeddings) yields strong empirical gains (e.g., validation loss dropping from ~2.4 to ~1.48) on Shakespeare text.

A practical NanoGPT workflow shows how to structure code (model.py, train.py), implement generation, and experiment with depth, heads, dropout, and layer norms.

Pre‑training on large corpora creates a general language model; alignment/fine‑tuning (e.g., RLHF) is a separate stage used to steer models toward helpful, safe assistant behavior.

INTRO TO GPT AND LANGUAGE MODELS

The talk begins by framing GPT as a language model that completes sequences of text. It emphasizes the probabilistic nature of outputs: the same prompt can yield different but plausible continuations. GPT is described as a decoder‑only Transformer—pre‑trained on vast data and then used to generate next tokens. A hands‑on example in the lecture uses a short prompt about AI to showcase how such models produce coherent, world‑changing prose.

TRANSFORMER FOUNDATIONS AND THE ATTENTION MECHANISM

The foundational idea is the Transformer architecture introduced in Attention Is All You Need (2017). The model uses attention to weigh relationships between tokens. The speaker explains the core components—queries, keys, values—and how the attention scores are computed, scaled, and masked to enforce autoregressive behavior. Residual connections and layer normalization are highlighted as critical for training stability, especially as networks deepen and context grows.

DATA PREP: SHAKESPEARE AND TOKENIZATION CHOICES

To make the concept concrete, a tiny Shakespeare dataset (~1 MB) is used. The data is treated at the character level, yielding a vocabulary of 65 characters and enabling straightforward encoding/decoding. The data is split 90/10 into train and validation sets, and training samples are formed by sampling short, overlapping blocks (e.g., block size 8) to create multiple training examples from each snippet, showcasing the need for efficient batching and context handling.

FROM SCRATCH TO NANOGPT: IMPLEMENTATION OVERVIEW

The lecture transitions from a minimal embedding‑based baseline to a Transformer by introducing NanogPT, a compact two‑file codebase (model.py and train.py). Early on a simple token embedding with a linear head is used; generation is performed by sampling from the last token. The narrative then introduces self‑attention, multi‑head setups, and the practical coding steps required to structure data, build the model, and run iterative improvements.

TRAINING LOOP AND LOSS: GETTING A MODEL TO LEARN

The training loop uses batches of chunked data, computes cross‑entropy loss, and updates parameters with an optimizer (Adam). The talk demonstrates that initial losses are relatively high (around 4+ in log space) and gradually decrease as the model learns. Early generation attempts yield random text, illustrating the progression from a toy model to something that begins to capture Shakespearean style as training proceeds.

SELF-ATTENTION DETAILS: QKV, MASKING, SCALE, AND HEADS

A core portion explains self‑attention in detail: each token produces a query, key, and value. The attention scores are computed as a dot product of queries with keys, masked to prevent looking at future tokens, and scaled by the square root of the head size to stabilize softmax. The resulting weights are used to combine values, enabling data‑dependent information flow between tokens. The section also covers single‑head vs multi‑head attention and why multiple heads improve expressivity.

BUILDING A DECODED-ONLY TRANSFORMER AND LARGER DEPTH

The speaker builds a decoder‑only Transformer with residual connections and layer normalization. Dropout is introduced to regularize training as depth increases. The pre‑norm (layer norm before the block) variant is highlighted as beneficial for stability in deeper networks. This section emphasizes architectural choices that mirror the original GPT design while adapting them for an experimental, educational setting.

SCALE, REGULARIZATION, AND REAL‑WORLD TRAINING RESULTS

As depth, heads, and embedding sizes increase, the model’s performance improves. The talk documents pushing a Shakespeare model to longer contexts (block size 256) and richer representations (embedding 384, 6 heads, 6 layers) with dropout. A validation loss of about 1.48 is reported after substantial training on an A100 GPU, demonstrating practical gains from scaling, regularization, and careful optimization.

PRE‑TRAINING VS FINE‑TUNING AND ALIGNMENT OVERVIEW

The session culminates with a high‑level map of how real GPT systems are built: (1) pre‑training on large, general corpora to learn broad language structure, and (2) fine‑tuning/alignment stages (e.g., supervised fine‑tuning, reward modeling, RLHF) to steer outputs toward helpful, safe assistant behavior. The speaker notes that the lecture focuses on pre‑training concepts and outlines the broad workflow without diving into proprietary alignment data or policies.

Mentioned in This Episode

●Software & Apps

●Tools

●Companies

●Studies Cited

●Concepts

●People Referenced

Minimal steps to build & train a tiny GPT (cheat sheet)

Practical takeaways from this episode

Do This

Do start with a small dataset (tiny Shakespeare) and a tiny character-level tokenizer to learn the pipeline quickly.

Do chunk data into fixed 'block_size' contexts and sample random batches (batch_size × block_size) for training.

Do use embeddings + positional embeddings, and implement scaled dot-product attention (queries, keys, values).

Do add residual (skip) connections and LayerNorm (pre-norm) to stabilize training of stacked blocks.

Do use Adam with a moderate learning rate and an evaluation routine that estimates train/val loss across multiple batches.

Do scale gradually: increase embedding size, number of heads / layers, and block size; add dropout if overfitting.

Avoid This

Don't try to train a large GPT-scale model on a CPU — use a GPU and smaller model config for feasibility.

Don't forget to mask future tokens in decoder/self-attention for autoregressive language modeling.

Don't omit LayerNorm / residuals for deeper stacks — optimization will become unstable without them.

Don't evaluate using a single noisy batch; average losses across many batches for a reliable signal.

Example model scaling and validation loss (from lecture)

Data extracted from this episode

Model config (embed / layers / heads / block)	Training time (approx)	Validation loss (approx)
Small baseline (nbed ~32, small blocks)	minutes	≈ 2.5
Intermediate (with single head, FFN)	minutes	≈ 2.06
Scaled model (embed=384, n_layer=6, n_head=6, block=256)	≈ 15 minutes on A100	≈ 1.48

Common Questions

The lecture trains a character-level Transformer on the tiny Shakespeare dataset (a ~1 MB concatenation of Shakespeare's works). See the dataset introduction at 231 seconds.