All Gather

Concept

A collective communication operation where each rank sends its data to all other ranks, resulting in each rank holding the complete gathered data.

Mentioned in 1 video

Videos Mentioning All Gather

Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 7: Parallelism

Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 7: Parallelism

Stanford Online

A collective communication operation where each rank sends its data to all other ranks, resulting in each rank holding the complete gathered data.