EUVIMEDEUVIMED entdecken
Crossref

SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget

Kun Wang, Jiani Cao, Zimu Zhou, Zhenjiang Li

IEEE Transactions on Mobile Computing · 2024-09 · 23 (9) · 8935-8950

DOI: 10.1109/tmc.2024.3355764 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2024-09. Verfasst von Kun Wang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 58 Literaturangaben. Gefördert von: Research Grants Council of Hong Kong, CityU SRG-Fd, Aromatic Plant Research Center.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Executing deep neural networks (DNNs) on edge artificial intelligence (AI) devices enables various autonomous mobile computing applications. However, the memory budget of edge AI devices restricts the number and complexity of DNNs allowed in such applications. Existing solutions, such as model compression or cloud offloading, reduce the memory footprint of DNN inference at the cost of decreased model accuracy or autonomy. To avoid these drawbacks, we divide DNN into blocks and swap them in and out in order, such that large DNNs can execute within a small memory budget. Nevertheless, naive swapping on edge AI devices induces significant delays due to the redundant memory operations in the DNN development ecosystem for edge AI devices. To this end, we develop SwapNet, an efficient DNN block swapping middleware for edge AI devices. We systematically eliminate the unnecessary memory operations during block swapping while retaining compatible with the deep learning frameworks, GPU backends, and hardware architectures of edge AI devices. We further showcase the utility of SwapNet via a multi-DNN scheduling scheme. Evaluations on eleven DNN inference tasks in three applications demonstrate that SwapNet achieves almost the same latency as the case with sufficient memory even when DNNs demand $2.32\times \,\sim$ 2 . 32 × ∼ $5.81\times$ 5 . 81 × memory beyond the available budget. The design of SwapNet also provides novel and feasible insights for deploying large language models (LLMs) on edge AI devices in the future.

Förderung

  • Research Grants Council of Hong Kong
  • CityU SRG-Fd
  • Aromatic Plant Research Center
  • City University of Hong Kong

Zitieren

APA

Wang, K., Cao, J., Zhou, Z., & Li, Z. (2024). SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget. IEEE Transactions on Mobile Computing, 23(9), 8935-8950. https://doi.org/10.1109/tmc.2024.3355764

Vancouver

Wang K, Cao J, Zhou Z, Li Z. SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget. IEEE Transactions on Mobile Computing. 2024;23(9):8935-8950. doi:10.1109/tmc.2024.3355764

BibTeX
@article{Wang2024,
  author = {Wang, Kun and Cao, Jiani and Zhou, Zimu and Li, Zhenjiang},
  title = {SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget},
  journal = {IEEE Transactions on Mobile Computing},
  year = {2024},
  volume = {23},
  number = {9},
  pages = {8935--8950},
  doi = {10.1109/tmc.2024.3355764}
}
Kostenlos starten →