What is Model distillation and Why Companies are doing it

Model distillation moves knowledge from a large, accurate model into a smaller, faster one. The big model plays "teacher," the compact model plays "student," and the student learns to copy the teacher's outputs while running on a fraction of the compute. You keep most of the accuracy and pay far…