All to All

Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 7: Parallelism

Stanford Online

A collective communication operation where each rank sends a specific message to every other rank, useful for dynamic routing and expert parallelism.