Distributed Data Parallel

Concept

A PyTorch implementation of data parallelism where gradients are averaged across all processes after the backward pass.

Mentioned in 1 video

Videos Mentioning Distributed Data Parallel

Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 7: Parallelism

Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 7: Parallelism

Stanford Online

A PyTorch implementation of data parallelism where gradients are averaged across all processes after the backward pass.