Knowledge Distillation

Concept

A technique where a smaller model (draft model) is trained to mimic the predictions of a larger, more powerful model (teacher model), helping to align their token distributions for speculative decoding.

Mentioned in 1 video

Videos Mentioning Knowledge Distillation

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 5 - LLM Systems

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 5 - LLM Systems

Stanford Online

A technique where a smaller model (draft model) is trained to mimic the predictions of a larger, more powerful model (teacher model), helping to align their token distributions for speculative decoding.