EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

The Role of Teacher Calibration in Knowledge Distillation

Suyoung Kim, Seonguk Park, Junhoo Lee, Nojun Kwak

IEEE Access · 2025 · 13 · 115548-115557

DOI: 10.1109/access.2025.3585106 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2025. Verfasst von Suyoung Kim und 3 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea, Korean Government, Institute for Information and communications Technology Planning and Evaluation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge Distillation (KD) has emerged as an effective model compression technique in deep learning, enabling the transfer of knowledge from a large teacher model to a compact student model. While KD has demonstrated significant success, it is not yet fully understood which factors contribute to improving the student’s performance. In this paper, we reveal a strong correlation between the teacher’s calibration error and the student’s accuracy. Therefore, we claim that the calibration of the teacher model is an important factor for effective KD. Furthermore, we demonstrate that the performance of KD can be improved by simply employing a calibration method that reduces the teacher’s calibration error. Our algorithm is versatile, demonstrating effectiveness across various tasks from classification to detection. Moreover, it can be easily integrated with existing state-of-the-art methods, consistently achieving superior performance.

Förderung

  • National Research Foundation of Korea
  • Korean Government
  • Institute for Information and communications Technology Planning and Evaluation

Zitieren

APA

Kim, S., Park, S., Lee, J., & Kwak, N. (2025). The Role of Teacher Calibration in Knowledge Distillation. IEEE Access, 13, 115548-115557. https://doi.org/10.1109/access.2025.3585106

Vancouver

Kim S, Park S, Lee J, Kwak N. The Role of Teacher Calibration in Knowledge Distillation. IEEE Access. 2025;13:115548-115557. doi:10.1109/access.2025.3585106

BibTeX
@article{Kim2025,
  author = {Kim, Suyoung and Park, Seonguk and Lee, Junhoo and Kwak, Nojun},
  title = {The Role of Teacher Calibration in Knowledge Distillation},
  journal = {IEEE Access},
  year = {2025},
  volume = {13},
  pages = {115548--115557},
  doi = {10.1109/access.2025.3585106}
}
Kostenlos starten →