Sparse Upcycling

BookMentioned in 1 video

A paper on which Yi Tay advised, focusing on vision with some T5 experiments, exploring the early stages of sparse upcycling which later became relevant for MoE models.

Videos Mentioning Sparse Upcycling

The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

Latent Space

A paper on which Yi Tay advised, focusing on vision with some T5 experiments, exploring the early stages of sparse upcycling which later became relevant for MoE models.