KL Divergence

Concept

A metric used in knowledge distillation to measure the difference between two probability distributions. It's employed to train smaller models to approximate the output distribution of larger models.

Mentioned in 3 videos

Videos Mentioning KL Divergence

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Latent Space

A metric used in knowledge distillation to measure the difference between two probability distributions. It's employed to train smaller models to approximate the output distribution of larger models.

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Latent Space

A distributional distance used as a constraint in RLHF objectives, acting as a guardrail to prevent overfitting to small datasets and maintaining model stability.

Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)

Lex Fridman

Used in variational learning to measure the difference between an approximating distribution (recognition model) and the true posterior.