QK clip

Software / App

A method used in optimizers to rescale parameters to prevent exploding gradients and attention logits, particularly in early training steps.

Mentioned in 1 video

Videos Mentioning QK clip

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

Latent Space

A method used in optimizers to rescale parameters to prevent exploding gradients and attention logits, particularly in early training steps.