SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget
IEEE Transactions on Mobile Computing · 2024-09 · 23 (9) · 8935-8950
DOI: 10.1109/tmc.2024.3355764 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2024-09. Verfasst von Kun Wang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 58 Literaturangaben. Gefördert von: Research Grants Council of Hong Kong, CityU SRG-Fd, Aromatic Plant Research Center.
Abstract (Originalsprache)
Executing deep neural networks (DNNs) on edge artificial intelligence (AI) devices enables various autonomous mobile computing applications. However, the memory budget of edge AI devices restricts the number and complexity of DNNs allowed in such applications. Existing solutions, such as model compression or cloud offloading, reduce the memory footprint of DNN inference at the cost of decreased model accuracy or autonomy. To avoid these drawbacks, we divide DNN into blocks and swap them in and out in order, such that large DNNs can execute within a small memory budget. Nevertheless, naive swapping on edge AI devices induces significant delays due to the redundant memory operations in the DNN development ecosystem for edge AI devices. To this end, we develop SwapNet, an efficient DNN block swapping middleware for edge AI devices. We systematically eliminate the unnecessary memory operations during block swapping while retaining compatible with the deep learning frameworks, GPU backends, and hardware architectures of edge AI devices. We further showcase the utility of SwapNet via a multi-DNN scheduling scheme. Evaluations on eleven DNN inference tasks in three applications demonstrate that SwapNet achieves almost the same latency as the case with sufficient memory even when DNNs demand $2.32\times \,\sim$ 2 . 32 × ∼ $5.81\times$ 5 . 81 × memory beyond the available budget. The design of SwapNet also provides novel and feasible insights for deploying large language models (LLMs) on edge AI devices in the future.
Förderung
- Research Grants Council of Hong Kong
- CityU SRG-Fd
- Aromatic Plant Research Center
- City University of Hong Kong
Zitieren
APA
Wang, K., Cao, J., Zhou, Z., & Li, Z. (2024). SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget. IEEE Transactions on Mobile Computing, 23(9), 8935-8950. https://doi.org/10.1109/tmc.2024.3355764
Vancouver
Wang K, Cao J, Zhou Z, Li Z. SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget. IEEE Transactions on Mobile Computing. 2024;23(9):8935-8950. doi:10.1109/tmc.2024.3355764
BibTeX
@article{Wang2024,
author = {Wang, Kun and Cao, Jiani and Zhou, Zimu and Li, Zhenjiang},
title = {SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget},
journal = {IEEE Transactions on Mobile Computing},
year = {2024},
volume = {23},
number = {9},
pages = {8935--8950},
doi = {10.1109/tmc.2024.3355764}
}
EUVIMED