Model compression using knowledge distillation with integrated gradients
Neural Computing and Applications · 2026-06 · 38 (11)
DOI: 10.1007/s00521-026-12137-5 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neural Computing and Applications, online erschienen am 28.05.2026. Verfasst von David E. Hernandez und 2 weiteren Autor:innen. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Model compression is critical for deploying deep learning models on resource-constrained devices. We introduce a novel method enhancing knowledge distillation with Integrated Gradients (IG) as a data augmentation strategy. Our approach overlays IG maps onto input images during training, providing student models with deeper insights into teacher models’ decision-making processes. Extensive evaluation on CIFAR-10 demonstrates that our IG-augmented knowledge distillation achieves 92.6% testing accuracy with a 4.1x compression factor–a significant 1.1 percentage point improvement (p < 0.001) over non-distilled models (91.5%). This compression reduces inference time from 140ms to 13ms. Our method precomputes IG maps before training, transforming substantial runtime costs into a one-time preprocessing step. We validate our approach through systematic ablation studies with attention transfer, comprehensive compression factor analysis (2.2 $$\times$$ –1122 $$\times$$ ), Monte Carlo simulations, and cross-dataset evaluation on ImageNet subsets, demonstrating that IG-based knowledge distillation consistently outperforms conventional approaches. Our results establish this framework as a viable compression technique for real-world deployment on edge devices while maintaining competitive accuracy.
Zitieren
APA
Hernandez, D. E., Chang, J. R., & Nordling, T. E. M. (2026). Model compression using knowledge distillation with integrated gradients. Neural Computing and Applications, 38(11). https://doi.org/10.1007/s00521-026-12137-5
Vancouver
Hernandez DE, Chang JR, Nordling TEM. Model compression using knowledge distillation with integrated gradients. Neural Computing and Applications. 2026;38(11). doi:10.1007/s00521-026-12137-5
BibTeX
@article{Hernandez2026,
author = {Hernandez, David E. and Chang, Jose Ramon and Nordling, Torbjörn E. M.},
title = {Model compression using knowledge distillation with integrated gradients},
journal = {Neural Computing and Applications},
year = {2026},
volume = {38},
number = {11},
doi = {10.1007/s00521-026-12137-5}
}
EUVIMED