FlashAttention
Software / App
Mentioned as an example of specialized kernels for Transformers, paralleled by efforts like FlashFFTConv for SSMs.
Mentioned in 4 videos
Videos Mentioning FlashAttention
![2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]](https://i.ytimg.com/vi/LPe6iC73lrc/maxresdefault.jpg)
2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Latent Space
Mentioned as an example of specialized kernels for Transformers, paralleled by efforts like FlashFFTConv for SSMs.
A Comprehensive Overview of Large Language Models - Latent Space Paper Club
Latent Space
An optimized implementation of the attention mechanism that improves memory efficiency.
llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE
Latent Space
An optimized implementation of the attention mechanism for transformers, which llm.c utilizes for improved performance.
The End of Finetuning — with Jeremy Howard of Fast.ai
Latent Space
An optimization technique for attention mechanisms, mentioned as an example of innovations that could be made easier with better languages like Mojo.