SWIGLU

ConceptMentioned in 1 video

A feed-forward activation function from a single-author paper by Noam Shazeer that was initially obscure but gained popularity after its use in T5 1.1, and which Yi Tay considers very effective.

Videos Mentioning SWIGLU

The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

Latent Space

A feed-forward activation function from a single-author paper by Noam Shazeer that was initially obscure but gained popularity after its use in T5 1.1, and which Yi Tay considers very effective.