linear attention
Concept
An early attempt (around 2020) to make attention mechanisms sub-quadratic by removing the softmax nonlinearity, but facing quality and hardware efficiency issues.
Mentioned in 1 video
An early attempt (around 2020) to make attention mechanisms sub-quadratic by removing the softmax nonlinearity, but facing quality and hardware efficiency issues.
![2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]](https://i.ytimg.com/vi/LPe6iC73lrc/maxresdefault.jpg)