EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Model compression using knowledge distillation with integrated gradients

David E. Hernandez, Jose Ramon Chang, Torbjörn E. M. Nordling

Neural Computing and Applications · 2026-06 · 38 (11)

DOI: 10.1007/s00521-026-12137-5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Neural Computing and Applications, online erschienen am 28.05.2026. Verfasst von David E. Hernandez und 2 weiteren Autor:innen. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Model compression is critical for deploying deep learning models on resource-constrained devices. We introduce a novel method enhancing knowledge distillation with Integrated Gradients (IG) as a data augmentation strategy. Our approach overlays IG maps onto input images during training, providing student models with deeper insights into teacher models’ decision-making processes. Extensive evaluation on CIFAR-10 demonstrates that our IG-augmented knowledge distillation achieves 92.6% testing accuracy with a 4.1x compression factor–a significant 1.1 percentage point improvement (p < 0.001) over non-distilled models (91.5%). This compression reduces inference time from 140ms to 13ms. Our method precomputes IG maps before training, transforming substantial runtime costs into a one-time preprocessing step. We validate our approach through systematic ablation studies with attention transfer, comprehensive compression factor analysis (2.2 $$\times$$ –1122 $$\times$$ ), Monte Carlo simulations, and cross-dataset evaluation on ImageNet subsets, demonstrating that IG-based knowledge distillation consistently outperforms conventional approaches. Our results establish this framework as a viable compression technique for real-world deployment on edge devices while maintaining competitive accuracy.

Zitieren

APA

Hernandez, D. E., Chang, J. R., & Nordling, T. E. M. (2026). Model compression using knowledge distillation with integrated gradients. Neural Computing and Applications, 38(11). https://doi.org/10.1007/s00521-026-12137-5

Vancouver

Hernandez DE, Chang JR, Nordling TEM. Model compression using knowledge distillation with integrated gradients. Neural Computing and Applications. 2026;38(11). doi:10.1007/s00521-026-12137-5

BibTeX
@article{Hernandez2026,
  author = {Hernandez, David E. and Chang, Jose Ramon and Nordling, Torbjörn E. M.},
  title = {Model compression using knowledge distillation with integrated gradients},
  journal = {Neural Computing and Applications},
  year = {2026},
  volume = {38},
  number = {11},
  doi = {10.1007/s00521-026-12137-5}
}
Kostenlos starten →