Efficient Layer-Granularity Unloading for LLMs in Edge Computing
IEEE Transactions on Mobile Computing · 2026-09 · 25 (9) · 15221-15233
DOI: 10.1109/tmc.2026.3697118 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2026-09. Verfasst von Zhenzheng Li und 4 weiteren Autor:innen. Die Arbeit nennt 42 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Guangdong Key Lab of AI and Multi-modal Data Processing, Beijing Normal-Hong Kong Baptist University.
Abstract (Originalsprache)
Advancements in edge computing and container technology have made it increasingly popular and convenient to deploy Large Language Models (LLMs) through containers at the edge. However, the limited GPU resources of edge servers make it impractical to retain the model in GPU memory for long periods due to the high memory cost, especially when they remain idle without user requests. Existing work unloads the entire idle models to reduce memory costs on edge servers, but reloading them introduces significant loading delays that affect task Quality of Service (QoS). Therefore, efficient management of idle models is a critical issue that has been largely neglected in existing research and requires urgent attention. To address this gap, this paper studies the problem of idle model management from the perspective of the trade-off between memory cost and loading delay under the QoS constraint. A novel layer-granularity model unloading method is proposed, which leverages the layered characteristics of the model. We formulate an online joint optimization problem to determine which layers to unload and when, and present a layer-granularity unloading strategy inspired by the ski rental problem to solve it. We implement a real system with layer-granularity unloading for LLMs on NVIDIA GPUs and validate the effectiveness of the proposed method. Experimental results show it effectively trades off memory cost and loading delay, improving overall performance by up to 39.6%.
Förderung
- National Natural Science Foundation of China
- Guangdong Key Lab of AI and Multi-modal Data Processing
- Beijing Normal-Hong Kong Baptist University
- Department of Education of Guangdong Province
- Institute of Artificial Intelligence and Future Networks and Engineering Center of AI and Future Education
- Guangdong Provincial Department of Science and Technology
- Zhuhai Science-Tech Innovation Bureau
- Interdisciplinary Intelligence Super Computer Center of Beijing Normal University at Zhuhai
Zitieren
APA
Li, Z., Tang, Z., Guo, J., Jia, W., & Zhao, W. (2026). Efficient Layer-Granularity Unloading for LLMs in Edge Computing. IEEE Transactions on Mobile Computing, 25(9), 15221-15233. https://doi.org/10.1109/tmc.2026.3697118
Vancouver
Li Z, Tang Z, Guo J, Jia W, Zhao W. Efficient Layer-Granularity Unloading for LLMs in Edge Computing. IEEE Transactions on Mobile Computing. 2026;25(9):15221-15233. doi:10.1109/tmc.2026.3697118
BibTeX
@article{Li2026,
author = {Li, Zhenzheng and Tang, Zhiqing and Guo, Jianxiong and Jia, Weijia and Zhao, Wei},
title = {Efficient Layer-Granularity Unloading for LLMs in Edge Computing},
journal = {IEEE Transactions on Mobile Computing},
year = {2026},
volume = {25},
number = {9},
pages = {15221--15233},
doi = {10.1109/tmc.2026.3697118}
}
EUVIMED