EUVIMEDTutustu EUVIMEDiin
Crossref

WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching

Xiangchen Li, Jiakun Fan, Qingyuan Wang, Dimitrios Spatharakis, Saeid Ghafouri, Hans Vandierendonck, Deepu John, Bo Ji, Ali R. Butt, Dimitrios Nikolopoulos

Proceedings of the ACM on Measurement and Analysis of Computing Systems · 2026-05-29 · 10 (2) · 1-31

DOI: 10.1145/3805653 ↗

At a glance

Journal article in Proceedings of the ACM on Measurement and Analysis of Computing Systems, published online on 2026-05-29. Written by Xiangchen Li and 9 co-authors. 44 references. Funded by: Virginia Tech College of Engineering, National Science Foundation.

TiivistelmäBibliografiset tiedotCiteRelated studies

Abstract (original language)

As Large Language Models (LLMs) become increasingly accessible to end users, an ever-growing number of inference requests are initiated from edge devices and computed on centralized GPU clusters. However, the resulting exponential growth in computation workload is placing significant strain on data centers, while edge devices remain largely underutilized, leading to imbalanced workloads and resource inefficiency across the network. Integrating edge devices into the LLM inference process via speculative decoding helps balance the workload between the edge and the cloud, while maintaining lossless prediction accuracy. In this paper, we identify and formalize two critical bottlenecks that limit the efficiency and scalability of distributed speculative LLM serving: Wasted Drafting Time and Verification Interference. To address these challenges, we propose WISP , an efficient and SLO-aware distributed LLM inference system that consists of an intelligent speculation controller, a verification time estimator, and a verification batch scheduler. These components collaboratively enhance drafting efficiency and optimize verification request scheduling on the server. Extensive numerical results show that WISP improves system capacity by up to 2.1× and 4.1×, and increases system goodput by up to 1.94× and 3.7×, compared to centralized serving and SLED , respectively.

Funding

  • Virginia Tech College of Engineering
  • National Science Foundation

Cite

APA

Li, X., Fan, J., Wang, Q., Spatharakis, D., Ghafouri, S., Vandierendonck, H., John, D., Ji, B., Butt, A. R., & Nikolopoulos, D. (2026). WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 10(2), 1-31. https://doi.org/10.1145/3805653

Vancouver

Li X, Fan J, Wang Q, Spatharakis D, Ghafouri S, Vandierendonck H, et al. WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching. Proceedings of the ACM on Measurement and Analysis of Computing Systems. 2026;10(2):1-31. doi:10.1145/3805653

BibTeX
@article{Li2026,
  author = {Li, Xiangchen and Fan, Jiakun and Wang, Qingyuan and Spatharakis, Dimitrios and Ghafouri, Saeid and Vandierendonck, Hans and John, Deepu and Ji, Bo and Butt, Ali R. and Nikolopoulos, Dimitrios},
  title = {WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching},
  journal = {Proceedings of the ACM on Measurement and Analysis of Computing Systems},
  year = {2026},
  volume = {10},
  number = {2},
  pages = {1--31},
  doi = {10.1145/3805653}
}
Start for free →