The Role of Teacher Calibration in Knowledge Distillation
IEEE Access · 2025 · 13 · 115548-115557
DOI: 10.1109/access.2025.3585106 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2025. Verfasst von Suyoung Kim und 3 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea, Korean Government, Institute for Information and communications Technology Planning and Evaluation.
Abstract (Originalsprache)
Knowledge Distillation (KD) has emerged as an effective model compression technique in deep learning, enabling the transfer of knowledge from a large teacher model to a compact student model. While KD has demonstrated significant success, it is not yet fully understood which factors contribute to improving the student’s performance. In this paper, we reveal a strong correlation between the teacher’s calibration error and the student’s accuracy. Therefore, we claim that the calibration of the teacher model is an important factor for effective KD. Furthermore, we demonstrate that the performance of KD can be improved by simply employing a calibration method that reduces the teacher’s calibration error. Our algorithm is versatile, demonstrating effectiveness across various tasks from classification to detection. Moreover, it can be easily integrated with existing state-of-the-art methods, consistently achieving superior performance.
Förderung
- National Research Foundation of Korea
- Korean Government
- Institute for Information and communications Technology Planning and Evaluation
Zitieren
APA
Kim, S., Park, S., Lee, J., & Kwak, N. (2025). The Role of Teacher Calibration in Knowledge Distillation. IEEE Access, 13, 115548-115557. https://doi.org/10.1109/access.2025.3585106
Vancouver
Kim S, Park S, Lee J, Kwak N. The Role of Teacher Calibration in Knowledge Distillation. IEEE Access. 2025;13:115548-115557. doi:10.1109/access.2025.3585106
BibTeX
@article{Kim2025,
author = {Kim, Suyoung and Park, Seonguk and Lee, Junhoo and Kwak, Nojun},
title = {The Role of Teacher Calibration in Knowledge Distillation},
journal = {IEEE Access},
year = {2025},
volume = {13},
pages = {115548--115557},
doi = {10.1109/access.2025.3585106}
}
EUVIMED