Q

QK clip

Tool / ProductMentioned in 1 video

A method used in optimizers to rescale parameters to prevent exploding gradients and attention logits, particularly in early training steps.

Videos Mentioning QK clip

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

Latent Space

A method used in optimizers to rescale parameters to prevent exploding gradients and attention logits, particularly in early training steps.