Megatron Core

[Paper Club] Upcycling Large Language Models into Mixture of Experts

Latent Space

A core component of Megatron-LM, providing optimized implementations for various LLM architectures and MoE.