Self-distillation

Inference, Diffusion, World Models, and More | YC Paper Club

Y Combinator

A form of distillation where a model distills knowledge into another model of the same size, surprisingly leading to further loss improvement.