EUVIMEDEUVIMED entdecken
Crossref

Exploring Direction Alignment and Discrepancy Standardization for Knowledge Distillation

Dingyao Chen, Xiao Teng, Xiang Zhang, Xun Yang, Long Lan

ACM Transactions on Knowledge Discovery from Data · 31.05.2026 · 20 (4) · 1-23

DOI: 10.1145/3803543 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Knowledge Discovery from Data, online erschienen am 20.04.2026. Verfasst von Dingyao Chen und 4 weiteren Autor:innen. Die Arbeit nennt 71 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Science and Technology Innovation Program of Hunan Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge Distillation (KD) is a widely popular model compression technique that can effectively transfer knowledge from a pre-trained, large-scale teacher model to a more compact and lightweight student model. Traditional KD methods aim to improve the student’s representation capability by mimicking the teacher’s features, e.g., minimizing the \(\mathcal{L}_{2}\) distance between their intermediate features. However, due to the capacity gap between the student and the teacher, student often struggles to precisely mimic the features of the teacher. To address this challenge, we propose to boost the knowledge distillation for the visual recognition tasks via Direction Alignment and Discrepancy Standardization ( DADS) , which exploits the feature scaling technique to distill from both the feature direction and feature discrepancy. To this end, we devise an efficient feature alignment module to align the dimensions of teacher and student features. Moreover, we align the direction of student features and teacher features, which are pre-processed by normalization. Furthermore, we leverage the Kullback–Leibler (KL) divergence to refine the features alignment, minimizing discrepancy in the distribution of features across samples, which is pre-processed by \(\mathcal{Z}\) -score standardization. In this way, our proposed approach can effectively transfer the knowledge from the teacher to the student, facilitating the downstream visual recognition applications, such as image classification and semantic segmentation. Extensive experimental analyses clearly validate the effectiveness of DADS . Compared with previous KD methods, our approach sets a new benchmark, achieving state-of-the-art results on visual recognition tasks.

Förderung

  • National Natural Science Foundation of China
  • Science and Technology Innovation Program of Hunan Province

Zitieren

APA

Chen, D., Teng, X., Zhang, X., Yang, X., & Lan, L. (2026). Exploring Direction Alignment and Discrepancy Standardization for Knowledge Distillation. ACM Transactions on Knowledge Discovery from Data, 20(4), 1-23. https://doi.org/10.1145/3803543

Vancouver

Chen D, Teng X, Zhang X, Yang X, Lan L. Exploring Direction Alignment and Discrepancy Standardization for Knowledge Distillation. ACM Transactions on Knowledge Discovery from Data. 2026;20(4):1-23. doi:10.1145/3803543

BibTeX
@article{Chen2026,
  author = {Chen, Dingyao and Teng, Xiao and Zhang, Xiang and Yang, Xun and Lan, Long},
  title = {Exploring Direction Alignment and Discrepancy Standardization for Knowledge Distillation},
  journal = {ACM Transactions on Knowledge Discovery from Data},
  year = {2026},
  volume = {20},
  number = {4},
  pages = {1--23},
  doi = {10.1145/3803543}
}
Kostenlos starten →