Switch Transformer

[Paper Club] Upcycling Large Language Models into Mixture of Experts

Latent Space

The first model to surpass one trillion model size and an early example of MoE architecture.