EUVIMEDEUVIMED entdecken
Crossref

Hierarchical Multi-Attention Transfer for Knowledge Distillation

Jianping Gou, Liyuan Sun, Baosheng Yu, Shaohua Wan, Dacheng Tao

ACM Transactions on Multimedia Computing, Communications, and Applications · 29.02.2024 · 20 (2) · 1-20

DOI: 10.1145/3568679 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 27.09.2023. Verfasst von Jianping Gou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 73-mal zitiert (Crossref) und nennt 55 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Qing Lan Project of Colleges and Universities of Jiangsu Province in 2020, Australian Research Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge distillation (KD) is a powerful and widely applicable technique for the compression of deep learning models. The main idea of knowledge distillation is to transfer knowledge from a large teacher model to a small student model, where the attention mechanism has been intensively explored in regard to its great flexibility for managing different teacher-student architectures. However, existing attention-based methods usually transfer similar attention knowledge from the intermediate layers of deep neural networks, leaving the hierarchical structure of deep representation learning poorly investigated for knowledge distillation. In this paper, we propose a hierarchical multi-attention transfer framework (HMAT) , where different types of attention are utilized to transfer the knowledge at different levels of deep representation learning for knowledge distillation. Specifically, position-based and channel-based attention knowledge characterize the knowledge from low-level and high-level feature representations, respectively, and activation-based attention knowledge characterize the knowledge from both mid-level and high-level feature representations. Extensive experiments on three popular visual recognition tasks, image classification, image retrieval, and object detection, demonstrate that the proposed hierarchical multi-attention transfer or HMAT significantly outperforms recent state-of-the-art KD methods.

Förderung

  • National Natural Science Foundation of China
  • Qing Lan Project of Colleges and Universities of Jiangsu Province in 2020
  • Australian Research Council

Zitieren

APA

Gou, J., Sun, L., Yu, B., Wan, S., & Tao, D. (2024). Hierarchical Multi-Attention Transfer for Knowledge Distillation. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(2), 1-20. https://doi.org/10.1145/3568679

Vancouver

Gou J, Sun L, Yu B, Wan S, Tao D. Hierarchical Multi-Attention Transfer for Knowledge Distillation. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(2):1-20. doi:10.1145/3568679

BibTeX
@article{Gou2024,
  author = {Gou, Jianping and Sun, Liyuan and Yu, Baosheng and Wan, Shaohua and Tao, Dacheng},
  title = {Hierarchical Multi-Attention Transfer for Knowledge Distillation},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2024},
  volume = {20},
  number = {2},
  pages = {1--20},
  doi = {10.1145/3568679}
}
Kostenlos starten →