EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Big2Small: Learning from masked image modelling with heterogeneous self‐supervised knowledge distillation

Ziming Wang, Shumin Han, Xiaodi Wang, Jing Hao, Xianbin Cao, Baochang Zhang

IET Cyber-Systems and Robotics · 2024-12 · 6 (4)

DOI: 10.1049/csy2.70002 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IET Cyber-Systems and Robotics, online erschienen am 31.12.2024. Verfasst von Ziming Wang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Small convolutional neural network (CNN)‐based models usually require transferring knowledge from a large model before they are deployed in computationally resource‐limited edge devices. Masked image modelling (MIM) methods achieve great success in various visual tasks but remain largely unexplored in knowledge distillation for heterogeneous deep models. The reason is mainly due to the significant discrepancy between the transformer‐based large model and the CNN‐based small network. In this paper, the authors develop the first heterogeneous self‐supervised knowledge distillation (HSKD) based on MIM, which can efficiently transfer knowledge from large transformer models to small CNN‐based models in a self‐supervised fashion. Our method builds a bridge between transformer‐based models and CNNs by training a UNet‐style student with sparse convolution, which can effectively mimic the visual representation inferred by a teacher over masked modelling. Our method is a simple yet effective learning paradigm to learn the visual representation and distribution of data from heterogeneous teacher models, which can be pre‐trained using advanced self‐supervised methods. Extensive experiments show that it adapts well to various models and sizes, consistently achieving state‐of‐the‐art performance in image classification, object detection, and semantic segmentation tasks. For example, in the Imagenet 1K dataset, HSKD improves the accuracy of Resnet‐50 (sparse) from 76.98% to 80.01%.

Zitieren

APA

Wang, Z., Han, S., Wang, X., Hao, J., Cao, X., & Zhang, B. (2024). Big2Small: Learning from masked image modelling with heterogeneous self‐supervised knowledge distillation. IET Cyber-Systems and Robotics, 6(4). https://doi.org/10.1049/csy2.70002

Vancouver

Wang Z, Han S, Wang X, Hao J, Cao X, Zhang B. Big2Small: Learning from masked image modelling with heterogeneous self‐supervised knowledge distillation. IET Cyber-Systems and Robotics. 2024;6(4). doi:10.1049/csy2.70002

BibTeX
@article{Wang2024,
  author = {Wang, Ziming and Han, Shumin and Wang, Xiaodi and Hao, Jing and Cao, Xianbin and Zhang, Baochang},
  title = {Big2Small: Learning from masked image modelling with heterogeneous self‐supervised knowledge distillation},
  journal = {IET Cyber-Systems and Robotics},
  year = {2024},
  volume = {6},
  number = {4},
  doi = {10.1049/csy2.70002}
}
Kostenlos starten →