EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Tailored knowledge distillation with automated loss function learning

Sheng Ran, Tao Huang, Wuyue Yang iD

PLOS One · 2025

DOI: 10.1371/journal.pone.0325599 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Sheng Ran und 2 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Statistics and Big Data, Renmin University of China, Beijing, China. Hauptthemen laut MEDLINE: Distillation, Machine Learning, Learning. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge Distillation (KD) is one of the most effective and widely used methods for model compression of large models. It has achieved significant success with the meticulous development of distillation losses. However, most state-of-the-art KD losses are manually crafted and task-specific, raising questions about their contribution to distillation efficacy. This paper unveils Learnable Knowledge Distillation (LKD), a novel approach that autonomously learns adaptive, performance-driven distillation losses. LKD revolutionizes KD by employing a bi-level optimization strategy and an iterative optimization that differentiably learns distillation losses aligned with the students’ validation loss. Building upon our proposed generic loss networks for logits and intermediate features, we derive a dynamic optimization strategy to adjust losses based on the student models’ changing states for enhanced performance and adaptability. Additionally, for a more robust loss, we introduce a uniform sampling of diverse previously-trained student models to train the loss with various convergence rates of predictions. With the more universally adaptable distillation framework of LKD, we conduct experiments on various datasets such as CIFAR and ImageNet, demonstrating our superior performance without the need for task-specific adjustments. For example, our LKD achieves 73.62% accuracy with the MobileNet model on ImageNet, significantly surpassing our KD baseline by 2.94%.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DistillationMachine LearningLearningHumansAlgorithmsNeural Networks, ComputerKnowledge

Autor:innen und Institutionen

  • Institute of Statistics and Big Data, Renmin University of China, Beijing, China.
  • Beijing Institute of Mathematical Sciences and Applications (BIMSA), Beijing, China.
  • The University of Sydney, Darlington, New South Wales, Australia.

ORCID-Profile: Wuyue Yang

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Ran, S., Huang, T., & Yang, W. (2025). Tailored knowledge distillation with automated loss function learning. PLOS One. https://doi.org/10.1371/journal.pone.0325599

Vancouver

Ran S, Huang T, Yang W. Tailored knowledge distillation with automated loss function learning. PLOS One. 2025. doi:10.1371/journal.pone.0325599

BibTeX
@article{Ran2025,
  author = {Ran, Sheng and Huang, Tao and Yang, Wuyue},
  title = {Tailored knowledge distillation with automated loss function learning},
  journal = {PLOS One},
  year = {2025},
  doi = {10.1371/journal.pone.0325599}
}
Kostenlos starten →