Hierarchical Multi-Attention Transfer for Knowledge Distillation
ACM Transactions on Multimedia Computing, Communications, and Applications · 29.02.2024 · 20 (2) · 1-20
DOI: 10.1145/3568679 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 27.09.2023. Verfasst von Jianping Gou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 73-mal zitiert (Crossref) und nennt 55 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Qing Lan Project of Colleges and Universities of Jiangsu Province in 2020, Australian Research Council.
Abstract (Originalsprache)
Knowledge distillation (KD) is a powerful and widely applicable technique for the compression of deep learning models. The main idea of knowledge distillation is to transfer knowledge from a large teacher model to a small student model, where the attention mechanism has been intensively explored in regard to its great flexibility for managing different teacher-student architectures. However, existing attention-based methods usually transfer similar attention knowledge from the intermediate layers of deep neural networks, leaving the hierarchical structure of deep representation learning poorly investigated for knowledge distillation. In this paper, we propose a hierarchical multi-attention transfer framework (HMAT) , where different types of attention are utilized to transfer the knowledge at different levels of deep representation learning for knowledge distillation. Specifically, position-based and channel-based attention knowledge characterize the knowledge from low-level and high-level feature representations, respectively, and activation-based attention knowledge characterize the knowledge from both mid-level and high-level feature representations. Extensive experiments on three popular visual recognition tasks, image classification, image retrieval, and object detection, demonstrate that the proposed hierarchical multi-attention transfer or HMAT significantly outperforms recent state-of-the-art KD methods.
Förderung
- National Natural Science Foundation of China
- Qing Lan Project of Colleges and Universities of Jiangsu Province in 2020
- Australian Research Council
Zitieren
APA
Gou, J., Sun, L., Yu, B., Wan, S., & Tao, D. (2024). Hierarchical Multi-Attention Transfer for Knowledge Distillation. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(2), 1-20. https://doi.org/10.1145/3568679
Vancouver
Gou J, Sun L, Yu B, Wan S, Tao D. Hierarchical Multi-Attention Transfer for Knowledge Distillation. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(2):1-20. doi:10.1145/3568679
BibTeX
@article{Gou2024,
author = {Gou, Jianping and Sun, Liyuan and Yu, Baosheng and Wan, Shaohua and Tao, Dacheng},
title = {Hierarchical Multi-Attention Transfer for Knowledge Distillation},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {2},
pages = {1--20},
doi = {10.1145/3568679}
}
EUVIMED