ML//training//knowledge distillation

2026-03-02

Knowledge distillation is a training method in which a small student model learns to mimic the behavior of a large teacher model, instead of being trained from scratch on raw data, and it is used mainly to compress a model for cheaper deployment.

Its key insight is that the teacher's softmax output distribution contains more information than hard labels. When GPT-4 gives 0.7 to cat, 0.2 to dog and 0.1 to tiger, that ranking teaches the student about the similarity between categories, and hard labels (1 for cat, 0 for everything else) throw it away.

The near-misses are the lesson. A teacher that gives 45 % to one continuation and 43 % to another is saying the two are almost interchangeable, which the winning token alone would hide. When the teacher's logits are not available (a closed model behind an API), distillation still works from its generated answers alone, as plain SFT on the teacher's outputs, losing that extra signal.

Temperature scaling amplifies these soft targets: raising the temperature during distillation smooths the distribution and makes the small differences between the non-top classes more visible to the student.

This is one route toward TinyML and mobile deployment. A smaller student can inherit selected behavior from a teacher, although retained capability and cost reduction depend on the task and training recipe. It differs from quantization, the other way to cut inference cost: quantization shrinks precision (float32 to int8) and leaves what the model knows, distillation shrinks the architecture (70B to 7B) and can change it.

At the edge it is one of three tools used together: quantization to int8, pruning of near-zero weights, and distillation of a large network trained on a GPU into one small enough for a Cortex-M microcontroller with a few hundred kilobytes of RAM or for the NPU of a smart camera. The student is usually trained on the teacher's outputs over the real sensor data, so it inherits the teacher's mistakes as well as its skill, and it must be validated on its own before it flies.

It is closely related to Chinchilla and scaling laws: once the optimal model size for a compute budget is known, distillation compresses a larger model down to that size after training.

Distillation also serves beyond compression: it is used to transfer capabilities across architectures. A Transformer teacher can distill into an SSM student, or a generalist into a specialist.

Synthetic data generation is a form of distillation: when a large model generates training data for a smaller model, the synthetic data carries the teacher's implicit knowledge.