EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Two-stage optimization based on heterogeneous branch fusion for knowledge distillation

Gang Li iD, Pengfei Lv, Yang Zhang iD, Chuanyun Xu, Zihan Ruan, Zheng Zhou, Xinyu Fan, Ru Wang, Pan He

PLOS One · 2025

DOI: 10.1371/journal.pone.0326711 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Gang Li und 8 weiteren Autor:innen. Beteiligte Einrichtung: School of Artificial Intelligence, Chongqing University of Technology, Chongqing, China. Hauptthemen laut MEDLINE: Knowledge, Distillation. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Chongqing Municipal Science and Technology Bureau, Chongqing Municipal Commission of Housing and Urban-Rural Development.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge distillation transfers knowledge from the teacher model to the student model, effectively improving the performance of the student model. However, relying solely on the fixed knowledge of the teacher model for guidance lacks the supplementation and expansion of knowledge, which limits the generalization ability of the student model. Therefore, this paper proposes two-stage optimization based on heterogeneous branch fusion for knowledge distillation (THFKD), which provides appropriate knowledge to the student model in different stages through a two-stage optimization strategy. Specifically, the pre-trained teacher offers stable and comprehensive static knowledge, preventing the student from deviating from the target early in the training process. Meanwhile, the student model acquires rich feature representations through heterogeneous branches and a progressive feature fusion module, generating dynamically updated collaborative learning objectives, thus effectively enhancing the diversity of dynamic knowledge. Finally, in the first stage, the ramp-up weight gradually increases the loss weight within the period, while in the second stage, consistent loss weights are applied. The two-stage optimization strategy fully exploits the advantages of each type of knowledge, thereby improving the generalization ability of the student model. Although no tests of statistical significance were carried out, our experimental results on standard datasets (CIFAR-100, Tiny-ImageNet) and long-tail datasets (CIFAR100-LT) suggest that THFKD may slightly improve the student model’s classification accuracy and generalization ability. For instance, using ResNet110-ResNet32 on the CIFAR-100 dataset, the accuracy reaches 75.41%, a 1.52% improvement over the state-of-the-art (SOTA).

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

KnowledgeDistillationHumansAlgorithmsLearning

Autor:innen und Institutionen

  • School of Artificial Intelligence, Chongqing University of Technology, Chongqing, China.
  • School of Computer and Information Science, Chongqing Normal University, Chongqing, China.

ORCID-Profile: Gang Li, Yang Zhang

Förderung

  • Chongqing Municipal Science and Technology Bureau
  • Chongqing Municipal Commission of Housing and Urban-Rural Development

Zitieren

APA

Li, G., Lv, P., Zhang, Y., Xu, C., Ruan, Z., Zhou, Z., Fan, X., Wang, R., & He, P. (2025). Two-stage optimization based on heterogeneous branch fusion for knowledge distillation. PLOS One. https://doi.org/10.1371/journal.pone.0326711

Vancouver

Li G, Lv P, Zhang Y, Xu C, Ruan Z, Zhou Z, et al. Two-stage optimization based on heterogeneous branch fusion for knowledge distillation. PLOS One. 2025. doi:10.1371/journal.pone.0326711

BibTeX
@article{Li2025,
  author = {Li, Gang and Lv, Pengfei and Zhang, Yang and Xu, Chuanyun and Ruan, Zihan and Zhou, Zheng and Fan, Xinyu and Wang, Ru and He, Pan},
  title = {Two-stage optimization based on heterogeneous branch fusion for knowledge distillation},
  journal = {PLOS One},
  year = {2025},
  doi = {10.1371/journal.pone.0326711}
}
Kostenlos starten →