EUVIMEDEUVIMED entdecken
Crossref

EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding

Daliang Xu, Wangsong Yin, Hao Zhang, Xin Jin, Ying Zhang, Shiyun Wei, Mengwei Xu, Xuanzhe Liu

IEEE Transactions on Mobile Computing · 2025-04 · 24 (4) · 3256-3273

DOI: 10.1109/tmc.2024.3513457 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2025-04. Verfasst von Daliang Xu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 62-mal zitiert (Crossref) und nennt 73 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Natural Science Fund for the Excellent Young Scientists Fund Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Generative tasks, such as text generation and question answering, are essential for mobile applications. Given their inherent privacy sensitivity, executing them on devices is demanded. Nowadays, the execution of these generative tasks heavily relies on the Large Language Models (LLMs). However, the scarce device memory severely hinders the scalability of these models. We present EdgeLLM, an efficient on-device LLM inference system for models whose sizes exceed the device's memory capacity. EdgeLLM is built atop speculative decoding, which delegates most tokens to a smaller, memory-resident (draft) LLM. EdgeLLM integrates three novel techniques: (1) Instead of generating a fixed width and depth token tree, EdgeLLM proposes compute-efficient branch navigation and verification to pace the progress of different branches according to their accepted probability to prevent the wasteful allocation of computing resources to the wrong branch and to verify them all at once efficiently. (2) It uses a self-adaptive fallback strategy that promptly initiates the verification process when the smaller LLM generates an incorrect token. (3) To not block the generation, EdgeLLM proposes speculatively generating tokens during large LLM verification with the compute-IO pipeline. Through extensive experiments, EdgeLLM exhibits impressive token generation speed which is up to 9.3× faster than existing engines.

Förderung

  • National Natural Science Foundation of China
  • National Natural Science Fund for the Excellent Young Scientists Fund Program

Zitieren

APA

Xu, D., Yin, W., Zhang, H., Jin, X., Zhang, Y., Wei, S., Xu, M., & Liu, X. (2025). EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding. IEEE Transactions on Mobile Computing, 24(4), 3256-3273. https://doi.org/10.1109/tmc.2024.3513457

Vancouver

Xu D, Yin W, Zhang H, Jin X, Zhang Y, Wei S, et al. EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding. IEEE Transactions on Mobile Computing. 2025;24(4):3256-3273. doi:10.1109/tmc.2024.3513457

BibTeX
@article{Xu2025,
  author = {Xu, Daliang and Yin, Wangsong and Zhang, Hao and Jin, Xin and Zhang, Ying and Wei, Shiyun and Xu, Mengwei and Liu, Xuanzhe},
  title = {EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding},
  journal = {IEEE Transactions on Mobile Computing},
  year = {2025},
  volume = {24},
  number = {4},
  pages = {3256--3273},
  doi = {10.1109/tmc.2024.3513457}
}
Kostenlos starten →