Exploring Direction Alignment and Discrepancy Standardization for Knowledge Distillation
ACM Transactions on Knowledge Discovery from Data · 31.05.2026 · 20 (4) · 1-23
DOI: 10.1145/3803543 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Knowledge Discovery from Data, online erschienen am 20.04.2026. Verfasst von Dingyao Chen und 4 weiteren Autor:innen. Die Arbeit nennt 71 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Science and Technology Innovation Program of Hunan Province.
Abstract (Originalsprache)
Knowledge Distillation (KD) is a widely popular model compression technique that can effectively transfer knowledge from a pre-trained, large-scale teacher model to a more compact and lightweight student model. Traditional KD methods aim to improve the student’s representation capability by mimicking the teacher’s features, e.g., minimizing the \(\mathcal{L}_{2}\) distance between their intermediate features. However, due to the capacity gap between the student and the teacher, student often struggles to precisely mimic the features of the teacher. To address this challenge, we propose to boost the knowledge distillation for the visual recognition tasks via Direction Alignment and Discrepancy Standardization ( DADS) , which exploits the feature scaling technique to distill from both the feature direction and feature discrepancy. To this end, we devise an efficient feature alignment module to align the dimensions of teacher and student features. Moreover, we align the direction of student features and teacher features, which are pre-processed by normalization. Furthermore, we leverage the Kullback–Leibler (KL) divergence to refine the features alignment, minimizing discrepancy in the distribution of features across samples, which is pre-processed by \(\mathcal{Z}\) -score standardization. In this way, our proposed approach can effectively transfer the knowledge from the teacher to the student, facilitating the downstream visual recognition applications, such as image classification and semantic segmentation. Extensive experimental analyses clearly validate the effectiveness of DADS . Compared with previous KD methods, our approach sets a new benchmark, achieving state-of-the-art results on visual recognition tasks.
Förderung
- National Natural Science Foundation of China
- Science and Technology Innovation Program of Hunan Province
Zitieren
APA
Chen, D., Teng, X., Zhang, X., Yang, X., & Lan, L. (2026). Exploring Direction Alignment and Discrepancy Standardization for Knowledge Distillation. ACM Transactions on Knowledge Discovery from Data, 20(4), 1-23. https://doi.org/10.1145/3803543
Vancouver
Chen D, Teng X, Zhang X, Yang X, Lan L. Exploring Direction Alignment and Discrepancy Standardization for Knowledge Distillation. ACM Transactions on Knowledge Discovery from Data. 2026;20(4):1-23. doi:10.1145/3803543
BibTeX
@article{Chen2026,
author = {Chen, Dingyao and Teng, Xiao and Zhang, Xiang and Yang, Xun and Lan, Long},
title = {Exploring Direction Alignment and Discrepancy Standardization for Knowledge Distillation},
journal = {ACM Transactions on Knowledge Discovery from Data},
year = {2026},
volume = {20},
number = {4},
pages = {1--23},
doi = {10.1145/3803543}
}
EUVIMED