Distillation means letting a small AI model learn from a large one so it becomes almost as good but faster and cheaper.
Example
A large model answers thousands of questions, and a small model learns from those answers.
For pros
The “student” model learns from the outputs or probabilities of the “teacher” model. Many small models are made this way. Note: many providers' terms forbid using their outputs to train competing models.
