DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy
Mathematics · 27.05.2024 · 12 (11) · 1672
DOI: 10.3390/math12111672 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematics, online erschienen am 27.05.2024. Verfasst von Xin Cheng und 4 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The complexity of deep neural network models (DNNs) severely limits their application on devices with limited computing and storage resources. Knowledge distillation (KD) is an attractive model compression technology that can effectively alleviate this problem. Multi-teacher knowledge distillation (MKD) aims to leverage the valuable and diverse knowledge distilled by multiple teacher networks to improve the performance of the student network. Existing approaches typically rely on simple methods such as averaging the prediction logits or using sub-optimal weighting strategies to fuse distilled knowledge from multiple teachers. However, employing these techniques cannot fully reflect the importance of teachers and may even mislead student’s learning. To address this issue, we propose a novel Decoupled Multi-Teacher Knowledge Distillation based on Entropy (DE-MKD). DE-MKD decouples the vanilla knowledge distillation loss and assigns adaptive weights to each teacher to reflect its importance based on the entropy of their predictions. Furthermore, we extend the proposed approach to distill the intermediate features from multiple powerful but cumbersome teachers to improve the performance of the lightweight student network. Extensive experiments on the publicly available CIFAR-100 image classification benchmark dataset with various teacher-student network pairs demonstrated the effectiveness and flexibility of our approach. For instance, the VGG8|ShuffleNetV2 model trained by DE-MKD reached 75.25%|78.86% top-one accuracy when choosing VGG13|WRN40-2 as the teacher, setting new performance records. In addition, surprisingly, the distilled student model outperformed the teacher in both teacher-student network pairs.
Zitieren
APA
Cheng, X., Zhang, Z., Weng, W., Yu, W., & Zhou, J. (2024). DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy. Mathematics, 12(11), 1672. https://doi.org/10.3390/math12111672
Vancouver
Cheng X, Zhang Z, Weng W, Yu W, Zhou J. DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy. Mathematics. 2024;12(11):1672. doi:10.3390/math12111672
BibTeX
@article{Cheng2024,
author = {Cheng, Xin and Zhang, Zhiqiang and Weng, Wei and Yu, Wenxin and Zhou, Jinjia},
title = {DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy},
journal = {Mathematics},
year = {2024},
volume = {12},
number = {11},
pages = {1672},
doi = {10.3390/math12111672}
}
EUVIMED