Tailored knowledge distillation with automated loss function learning
PLOS One · 2025
DOI: 10.1371/journal.pone.0325599 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Sheng Ran und 2 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Statistics and Big Data, Renmin University of China, Beijing, China. Hauptthemen laut MEDLINE: Distillation, Machine Learning, Learning. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Knowledge Distillation (KD) is one of the most effective and widely used methods for model compression of large models. It has achieved significant success with the meticulous development of distillation losses. However, most state-of-the-art KD losses are manually crafted and task-specific, raising questions about their contribution to distillation efficacy. This paper unveils Learnable Knowledge Distillation (LKD), a novel approach that autonomously learns adaptive, performance-driven distillation losses. LKD revolutionizes KD by employing a bi-level optimization strategy and an iterative optimization that differentiably learns distillation losses aligned with the students’ validation loss. Building upon our proposed generic loss networks for logits and intermediate features, we derive a dynamic optimization strategy to adjust losses based on the student models’ changing states for enhanced performance and adaptability. Additionally, for a more robust loss, we introduce a uniform sampling of diverse previously-trained student models to train the loss with various convergence rates of predictions. With the more universally adaptable distillation framework of LKD, we conduct experiments on various datasets such as CIFAR and ImageNet, demonstrating our superior performance without the need for task-specific adjustments. For example, our LKD achieves 73.62% accuracy with the MobileNet model on ImageNet, significantly surpassing our KD baseline by 2.94%.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Institute of Statistics and Big Data, Renmin University of China, Beijing, China.
- Beijing Institute of Mathematical Sciences and Applications (BIMSA), Beijing, China.
- The University of Sydney, Darlington, New South Wales, Australia.
ORCID-Profile: Wuyue Yang
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Ran, S., Huang, T., & Yang, W. (2025). Tailored knowledge distillation with automated loss function learning. PLOS One. https://doi.org/10.1371/journal.pone.0325599
Vancouver
Ran S, Huang T, Yang W. Tailored knowledge distillation with automated loss function learning. PLOS One. 2025. doi:10.1371/journal.pone.0325599
BibTeX
@article{Ran2025,
author = {Ran, Sheng and Huang, Tao and Yang, Wuyue},
title = {Tailored knowledge distillation with automated loss function learning},
journal = {PLOS One},
year = {2025},
doi = {10.1371/journal.pone.0325599}
}
EUVIMED