EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Knowledge distillation based on multi-layer fusion features

Shengyuan Tan iD, Rongzuo Guo, Jialiang Tang, Ning Jiang, Junying Zou

PLOS ONE · 2023

DOI: 10.1371/journal.pone.0285901 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS ONE, 2023. Verfasst von Shengyuan Tan und 4 weiteren Autor:innen. Beteiligte Einrichtung: College of Computer Science, Sichuan Normal Univeersity, Chengdu, Sichuan, 610101, China. Hauptthemen laut MEDLINE: Dietary Fiber, Educational Personnel. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Sichuan Province Science and Technology Support Program, National Natural Science Foundation of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge distillation improves the performance of a small student network by promoting it to learn the knowledge from a pre-trained high-performance but bulky teacher network. Generally, most of the current knowledge distillation methods extract relatively simple features from the middle or bottom layer of teacher network for knowledge transmission. However, the above methods ignore the fusion of features, and the fused features contain richer information. We believe that the richer and better information contained in the knowledge delivered by teachers to students, the easier it is for students to perform better. In this paper, we propose a new method called Multi-feature Fusion Knowledge Distillation (MFKD) to extract and utilize the expressive fusion features of teacher network. Specifically, we extract feature maps from different positions of the network, i.e., the middle layer, the bottom layer, and even the front layer of the network. To properly utilize these features, this method designs a multi-feature fusion scheme to integrate them together. Compared to features extracted from single location of teacher network, the final fusion feature map contains meaningful information. Extensive experiments on image classification tasks demonstrate that the student network trained by our MFKD can learn from the fusion features, leading to superior performance. The results show that MFKD can improve the Top-1 accuracy of ResNet20 and VGG8 by 1.82% and 3.35% respectively on the CIFAR-100 dataset, which is better than state-of-the-art many existing methods.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Dietary FiberEducational PersonnelHumansDistillationKnowledgeStudents

Substanzen

Dietary Fiber

Autor:innen und Institutionen

  • College of Computer Science, Sichuan Normal Univeersity, Chengdu, Sichuan, 610101, China.
  • School of Computer Science and Technology, Southwest University of Science and Technology, Mianyang, Sichuan, 621010, China.

ORCID-Profile: Shengyuan Tan

Förderung

  • Sichuan Province Science and Technology Support Program
  • National Natural Science Foundation of China

Zitieren

APA

Tan, S., Guo, R., Tang, J., Jiang, N., & Zou, J. (2023). Knowledge distillation based on multi-layer fusion features. PLOS ONE. https://doi.org/10.1371/journal.pone.0285901

Vancouver

Tan S, Guo R, Tang J, Jiang N, Zou J. Knowledge distillation based on multi-layer fusion features. PLOS ONE. 2023. doi:10.1371/journal.pone.0285901

BibTeX
@article{Tan2023,
  author = {Tan, Shengyuan and Guo, Rongzuo and Tang, Jialiang and Jiang, Ning and Zou, Junying},
  title = {Knowledge distillation based on multi-layer fusion features},
  journal = {PLOS ONE},
  year = {2023},
  doi = {10.1371/journal.pone.0285901}
}
Kostenlos starten →