EUVIMEDEUVIMED entdecken
Crossref

Research on Two-Stage Knowledge Distillation and Hybrid Compression of Convolutional Neural Networks to Reduce their Computational Complexity

Kozynets A, Demkiv L

Artificial Intelligence · 30.03.2026 · 31 (AI.2026.31(1)) · 59-69

DOI: 10.15407/jai2026.01.059 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Artificial Intelligence, erschienen am 30.03.2026. Verfasst von Kozynets A und 1 weiteren Autor:innen. Die Arbeit nennt 12 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The article considers the problem of effective knowledge transfer between deep convolutional neural networks of different capacities for their further deployment on hardware platforms with limited computing resources. The main problem of standard knowledge distillation protocols is the occurrence of "gradient shock" during the initialization of the student model on specific data sets, which leads to the destruction of the feature space and the loss of final accuracy. To overcome this limitation, the Two-Stage Distillation algorithm was developed and implemented. The proposed approach divides the learning process into the classifier stabilization phase and the deep distillation phase. Experimental research was conducted on the ResNet and VGG architectural families. The results obtained confirm that the use of the proposed algorithm allows to increase the accuracy of compact models by 1.5–2.6% compared to standard training. In addition, the work investigated and experimentally confirmed the phenomenon of "distillation recovery" - the ability of the algorithm to restore the accuracy of the model after aggressive structural pruning. It is proven that the use of "soft goals" of the teacher in a narrowed search space allows the sparse ResNet18 model to achieve an accuracy of 77.8%, which exceeds the basic full-size student model

Zitieren

APA

A, K., & L, D. (2026). Research on Two-Stage Knowledge Distillation and Hybrid Compression of Convolutional Neural Networks to Reduce their Computational Complexity. Artificial Intelligence, 31(AI.2026.31(1)), 59-69. https://doi.org/10.15407/jai2026.01.059

Vancouver

A K, L D. Research on Two-Stage Knowledge Distillation and Hybrid Compression of Convolutional Neural Networks to Reduce their Computational Complexity. Artificial Intelligence. 2026;31(AI.2026.31(1)):59-69. doi:10.15407/jai2026.01.059

BibTeX
@article{A2026,
  author = {A, Kozynets and L, Demkiv},
  title = {Research on Two-Stage Knowledge Distillation and Hybrid Compression of Convolutional Neural Networks to Reduce their Computational Complexity},
  journal = {Artificial Intelligence},
  year = {2026},
  volume = {31},
  number = {AI.2026.31(1)},
  pages = {59--69},
  doi = {10.15407/jai2026.01.059}
}
Kostenlos starten →