Sparse Upcycling

The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

Latent Space

A paper on which Yi Tay advised, focusing on vision with some T5 experiments, exploring the early stages of sparse upcycling which later became relevant for MoE models.