EUVIMEDEUVIMED entdecken
Crossref

EPSD: Early Pruning with Self-Distillation for Efficient Model Compression

Dong Chen, Ning Liu, Yichen Zhu, Zhengping Che, Rui Ma, Fachao Zhang, Xiaofeng Mou, Yi Chang, Jian Tang

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (10) · 11258-11266

DOI: 10.1609/aaai.v38i10.29004 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Dong Chen und 8 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Neural network compression techniques, such as knowledge distillation (KD) and network pruning, have received increasing attention. Recent work `Prune, then Distill' reveals that a pruned student-friendly teacher network can benefit the performance of KD. However, the conventional teacher-student pipeline, which entails cumbersome pre-training of the teacher and complicated compression steps, makes pruning with KD less efficient. In addition to compressing models, recent compression techniques also emphasize the aspect of efficiency. Early pruning demands significantly less computational cost in comparison to the conventional pruning methods as it does not require a large pre-trained model. Likewise, a special case of KD, known as self-distillation (SD), is more efficient since it requires no pre-training or student-teacher pair selection. This inspires us to collaborate early pruning with SD for efficient model compression. In this work, we propose the framework named Early Pruning with Self-Distillation (EPSD), which identifies and preserves distillable weights in early pruning for a given SD task. EPSD efficiently combines early pruning and self-distillation in a two-step process, maintaining the pruned network's trainability for compression. Instead of a simple combination of pruning and SD, EPSD enables the pruned network to favor SD by keeping more distillable weights before training to ensure better distillation of the pruned network. We demonstrated that EPSD improves the training of pruned networks, supported by visual and quantitative analyses. Our evaluation covered diverse benchmarks (CIFAR-10/100, Tiny-ImageNet, full ImageNet, CUB-200-2011, and Pascal VOC), with EPSD outperforming advanced pruning and SD techniques.

Zitieren

APA

Chen, D., Liu, N., Zhu, Y., Che, Z., Ma, R., Zhang, F., Mou, X., Chang, Y., & Tang, J. (2024). EPSD: Early Pruning with Self-Distillation for Efficient Model Compression. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10), 11258-11266. https://doi.org/10.1609/aaai.v38i10.29004

Vancouver

Chen D, Liu N, Zhu Y, Che Z, Ma R, Zhang F, et al. EPSD: Early Pruning with Self-Distillation for Efficient Model Compression. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(10):11258-11266. doi:10.1609/aaai.v38i10.29004

BibTeX
@article{Chen2024,
  author = {Chen, Dong and Liu, Ning and Zhu, Yichen and Che, Zhengping and Ma, Rui and Zhang, Fachao and Mou, Xiaofeng and Chang, Yi and Tang, Jian},
  title = {EPSD: Early Pruning with Self-Distillation for Efficient Model Compression},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {10},
  pages = {11258--11266},
  doi = {10.1609/aaai.v38i10.29004}
}
Kostenlos starten →