EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding
IEEE Transactions on Mobile Computing · 2025-04 · 24 (4) · 3256-3273
DOI: 10.1109/tmc.2024.3513457 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2025-04. Verfasst von Daliang Xu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 62-mal zitiert (Crossref) und nennt 73 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Natural Science Fund for the Excellent Young Scientists Fund Program.
Abstract (Originalsprache)
Generative tasks, such as text generation and question answering, are essential for mobile applications. Given their inherent privacy sensitivity, executing them on devices is demanded. Nowadays, the execution of these generative tasks heavily relies on the Large Language Models (LLMs). However, the scarce device memory severely hinders the scalability of these models. We present EdgeLLM, an efficient on-device LLM inference system for models whose sizes exceed the device's memory capacity. EdgeLLM is built atop speculative decoding, which delegates most tokens to a smaller, memory-resident (draft) LLM. EdgeLLM integrates three novel techniques: (1) Instead of generating a fixed width and depth token tree, EdgeLLM proposes compute-efficient branch navigation and verification to pace the progress of different branches according to their accepted probability to prevent the wasteful allocation of computing resources to the wrong branch and to verify them all at once efficiently. (2) It uses a self-adaptive fallback strategy that promptly initiates the verification process when the smaller LLM generates an incorrect token. (3) To not block the generation, EdgeLLM proposes speculatively generating tokens during large LLM verification with the compute-IO pipeline. Through extensive experiments, EdgeLLM exhibits impressive token generation speed which is up to 9.3× faster than existing engines.
Förderung
- National Natural Science Foundation of China
- National Natural Science Fund for the Excellent Young Scientists Fund Program
Zitieren
APA
Xu, D., Yin, W., Zhang, H., Jin, X., Zhang, Y., Wei, S., Xu, M., & Liu, X. (2025). EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding. IEEE Transactions on Mobile Computing, 24(4), 3256-3273. https://doi.org/10.1109/tmc.2024.3513457
Vancouver
Xu D, Yin W, Zhang H, Jin X, Zhang Y, Wei S, et al. EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding. IEEE Transactions on Mobile Computing. 2025;24(4):3256-3273. doi:10.1109/tmc.2024.3513457
BibTeX
@article{Xu2025,
author = {Xu, Daliang and Yin, Wangsong and Zhang, Hao and Jin, Xin and Zhang, Ying and Wei, Shiyun and Xu, Mengwei and Liu, Xuanzhe},
title = {EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding},
journal = {IEEE Transactions on Mobile Computing},
year = {2025},
volume = {24},
number = {4},
pages = {3256--3273},
doi = {10.1109/tmc.2024.3513457}
}
EUVIMED