KL Divergence
ConceptMentioned in 2 videos
A metric used in knowledge distillation to measure the difference between two probability distributions. It's employed to train smaller models to approximate the output distribution of larger models.
Videos Mentioning KL Divergence
![[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz](https://i.ytimg.com/vi/8BN9CdIYaqc/maxresdefault.jpg)
[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz
Latent Space
A metric used in knowledge distillation to measure the difference between two probability distributions. It's employed to train smaller models to approximate the output distribution of larger models.
The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert
Latent Space
A distributional distance used as a constraint in RLHF objectives, acting as a guardrail to prevent overfitting to small datasets and maintaining model stability.