linear attention
Concept
An early attempt (around 2020) to make attention mechanisms sub-quadratic by removing the softmax nonlinearity, but facing quality and hardware efficiency issues.
Mentioned in 2 videos
Save the 2 videos on linear attention to your own pod.
Sign up free to keep building your knowledge base on linear attention as more episodes are added.
Videos Mentioning linear attention
![2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]](https://i.ytimg.com/vi/LPe6iC73lrc/maxresdefault.jpg)
2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Latent Space
An early attempt (around 2020) to make attention mechanisms sub-quadratic by removing the softmax nonlinearity, but facing quality and hardware efficiency issues.
LLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan
Stanford Online
An optimization of the attention mechanism that reduces the quadratic complexity to linear, improving efficiency for long contexts.