TeDri:Teacher-Driven Region Knowledge Distillation
IEEE Transactions on Knowledge and Data Engineering · 2026-06 · 38 (6) · 3975-3990
DOI: 10.1109/tkde.2026.3675551 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Knowledge and Data Engineering, 2026-06. Verfasst von Zherui Zhang und 7 weiteren Autor:innen. Die Arbeit nennt 55 Literaturangaben. Gefördert von: National Key Research and Development Program of China, Beijing Natural Science Foundation, National Natural Science Foundation of China.
Abstract (Originalsprache)
Knowledge distillation as a practical tool to enhance the performance of small-capacity student networks on downstream tasks comes at the cost of a lengthy distillation process due to the online inference of teacher networks, especially when there is a large capacity gap between them. Therefore, in this paper, we propose a fast distillation framework called TeDri based on region images by offline saving relevant regional information and its teacher guidance. Specifically, first, to alleviate the lack of diversity caused by the fixed augmentation path in region images, we propose Teacher-driven MixUp strategies with mild intensity and advocate binding the mixing factor $\lambda$ λ with teacher guidance confidence, where more confident category representations dominate the MixUp process. Furthermore, recognizing the need to evaluate these randomly cropped regions, and we propose region contrastive learning, encourage the student network to mimic the region partitioning behavior of the teacher, promoting a comprehensive understanding of global semantic content from multiple local perspectives. Finally, we introduce region mutual learning, employing spatial constraints among regions to require the student network towards consistent content interpretation across localized regions. Experiments on CIFAR-100 and ImageNet-1 K validate the effectiveness of the proposed TeDri, achieving competitive performance while significantly reducing training time.
Förderung
- National Key Research and Development Program of China
- Beijing Natural Science Foundation
- National Natural Science Foundation of China
- Taishan Scholars Program
- Key R&D Program of Shandong Province, China
- Shandong Provincial University Youth Innovation and Technology Support Program
- Shandong Provincial Natural Science Foundation for Young Scholars Program
- Qilu University of Technology
Zitieren
APA
Zhang, Z., Wang, C., Xu, R., Xu, W., Pei, X., Xu, S., Xu, W., & Guo, L. (2026). TeDri:Teacher-Driven Region Knowledge Distillation. IEEE Transactions on Knowledge and Data Engineering, 38(6), 3975-3990. https://doi.org/10.1109/tkde.2026.3675551
Vancouver
Zhang Z, Wang C, Xu R, Xu W, Pei X, Xu S, et al. TeDri:Teacher-Driven Region Knowledge Distillation. IEEE Transactions on Knowledge and Data Engineering. 2026;38(6):3975-3990. doi:10.1109/tkde.2026.3675551
BibTeX
@article{Zhang2026,
author = {Zhang, Zherui and Wang, Changwei and Xu, Rongtao and Xu, Wenhao and Pei, Xingtian and Xu, Shibiao and Xu, Wenbo and Guo, Li},
title = {TeDri:Teacher-Driven Region Knowledge Distillation},
journal = {IEEE Transactions on Knowledge and Data Engineering},
year = {2026},
volume = {38},
number = {6},
pages = {3975--3990},
doi = {10.1109/tkde.2026.3675551}
}
EUVIMED