DDP

Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 8: Parallelism

Stanford Online

Naive Data Parallelism, used as a baseline for comparison with Zero stages, characterized by replicating model copies across GPUs.