Knowledge Distillation Meets Open-Set Semi-supervised Learning
International Journal of Computer Vision · 2025-01 · 133 (1) · 315-334
DOI: 10.1007/s11263-024-02192-7 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 26.07.2024. Verfasst von Jing Yang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Existing knowledge distillation methods mostly focus on distillation of teacher’s prediction and intermediate activation. However, the structured representation, which arguably is one of the most critical ingredients of deep models, is largely overlooked. In this work, we propose a novel semantic representational distillation (SRD) method dedicated for distilling representational knowledge semantically from a pretrained teacher to a target student. The key idea is that we leverage the teacher’s classifier as a semantic critic for evaluating the representations of both teacher and student and distilling the semantic knowledge with high-order structured information over all feature dimensions. This is accomplished by introducing a notion of cross-network logit computed through passing student’s representation into teacher’s classifier. Further, considering the set of seen classes as a basis for the semantic space in a combinatorial perspective, we scale SRD to unseen classes for enabling effective exploitation of largely available, arbitrary unlabeled training data. At the problem level, this establishes an interesting connection between knowledge distillation with open-set semi-supervised learning (SSL). Extensive experiments show that our SRD outperforms significantly previous state-of-the-art knowledge distillation methods on both coarse object classification and fine face recognition tasks, as well as less studied yet practically crucial binary network distillation. Under more realistic open-set SSL settings we introduce, we reveal that knowledge distillation is generally more effective than existing out-of-distribution sample detection, and our proposed SRD is superior over both previous distillation and SSL competitors. The source code is available at https://github.com/jingyang2017/SRD_ossl .
Zitieren
APA
Yang, J., Zhu, X., Bulat, A., Martinez, B., & Tzimiropoulos, G. (2025). Knowledge Distillation Meets Open-Set Semi-supervised Learning. International Journal of Computer Vision, 133(1), 315-334. https://doi.org/10.1007/s11263-024-02192-7
Vancouver
Yang J, Zhu X, Bulat A, Martinez B, Tzimiropoulos G. Knowledge Distillation Meets Open-Set Semi-supervised Learning. International Journal of Computer Vision. 2025;133(1):315-334. doi:10.1007/s11263-024-02192-7
BibTeX
@article{Yang2025,
author = {Yang, Jing and Zhu, Xiatian and Bulat, Adrian and Martinez, Brais and Tzimiropoulos, Georgios},
title = {Knowledge Distillation Meets Open-Set Semi-supervised Learning},
journal = {International Journal of Computer Vision},
year = {2025},
volume = {133},
number = {1},
pages = {315--334},
doi = {10.1007/s11263-024-02192-7}
}
EUVIMED