Knowledge Distillation
Also known as: Distillation
Knowledge distillation trains a smaller student to mimic a larger teacher’s outputs or hidden states, compressing capability into a cheaper model.
It complements quantization and pruning in the efficiency toolbox.