EUVIMEDEUVIMED entdecken
Crossref

Structured Directional Pruning via Perturbation Orthogonal Projection

Xiaofeng Liu, Qing Wang, Yunfeng Shao, Yanhui Geng, Yinchuan Li

IEEE Transactions on Signal Processing · 2024 · 72 · 5439-5453

DOI: 10.1109/tsp.2024.3501674 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Signal Processing, 2024. Verfasst von Xiaofeng Liu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 54 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Key Research and Development Program of Tibet Autonomous Region, Science and Technology Major Project of Tibetan Autonomous Region of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Despite the great potential of artificial intelligence (AI), which promotes machines to mimic human intelligence in performing tasks, it requires a deep/extensive model with a sufficient number of parameters to enhance the expressive ability. This aspect often hinders the application of AI on resource-constrained devices. Structured pruning is an effective compression technique that reduces the computation of neural networks. However, it typically achieves parameter reduction at the cost of non-negligible accuracy loss, necessitating fine-tuning. This paper introduces a novel technique called Structured Directional Pruning (SDP) and its fast solver, Alternating Structured Directional Pruning (AltSDP). SDP is a general energy-efficient coarse-grained pruning method that enables efficient model pruning without requiring fine-tuning or expert knowledge of the desired sparsity level. Theoretical analysis confirms that the fast solver, AltSDP, achieves SDP asymptotically after sufficient training. Experimental results validate that AltSDP reaches the same minimum valley as the vanilla optimizer, namely stochastic gradient descent (SGD), while maintaining a constant training loss. Additionally, AltSDP achieves state-of-the-art pruned accuracy integrating pruning into the initial training process without the need for fine-tuning. Consequently, the newly proposed SDP, along with its fast solver AltSDP, can significantly facilitate the development of shrinking deep neural networks (DNNs) and enable the deployment of AI on resource-constrained devices.

Förderung

  • National Natural Science Foundation of China
  • Key Research and Development Program of Tibet Autonomous Region
  • Science and Technology Major Project of Tibetan Autonomous Region of China

Zitieren

APA

Liu, X., Wang, Q., Shao, Y., Geng, Y., & Li, Y. (2024). Structured Directional Pruning via Perturbation Orthogonal Projection. IEEE Transactions on Signal Processing, 72, 5439-5453. https://doi.org/10.1109/tsp.2024.3501674

Vancouver

Liu X, Wang Q, Shao Y, Geng Y, Li Y. Structured Directional Pruning via Perturbation Orthogonal Projection. IEEE Transactions on Signal Processing. 2024;72:5439-5453. doi:10.1109/tsp.2024.3501674

BibTeX
@article{Liu2024,
  author = {Liu, Xiaofeng and Wang, Qing and Shao, Yunfeng and Geng, Yanhui and Li, Yinchuan},
  title = {Structured Directional Pruning via Perturbation Orthogonal Projection},
  journal = {IEEE Transactions on Signal Processing},
  year = {2024},
  volume = {72},
  pages = {5439--5453},
  doi = {10.1109/tsp.2024.3501674}
}
Kostenlos starten →