EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Optimal Scheduling Algorithms for LLM Inference: Theory and Practice

Agrim Bari, Parikshit Hegde, Gustavo de Veciana

ACM SIGMETRICS Performance Evaluation Review · 20.07.2026 · 54 (1) · 238-240

DOI: 10.1145/3820171.3806878 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM SIGMETRICS Performance Evaluation Review, online erschienen am 27.07.2026. Verfasst von Agrim Bari und 2 weiteren Autor:innen. Die Arbeit nennt 5 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the growing use of Large Language Model (LLM)-based tools like ChatGPT, Perplexity, and Gemini across industries, there is a rising need for efficient LLM inference systems. These systems handle requests with a unique two-phase computation structure: a prefill-phase that processes the full input prompt and a decode-phase that autoregressively generates tokens one at a time. This structure calls for new strategies for routing and scheduling requests. In this paper, we take a comprehensive approach to this challenge by developing a theoretical framework that models routing and scheduling in LLM inference systems. We identify two key design principles—optimal tiling and dynamic resource allocation—that are essential for achieving high throughput. Guided by these principles, we propose the Resource-Aware Dynamic (RAD) scheduler and prove that it achieves throughput optimality under mild conditions. To address practical Service Level Objectives (SLOs) such as serving requests with different Time Between Token (TBT) constraints, we design the SLO-Aware LLM Inference (SLAI) scheduler. SLAI uses real-time measurements to prioritize decode requests that are close to missing their TBT deadlines and reorders prefill requests based on known prompt lengths to further reduce the Time To First Token (TTFT) delays. We evaluate SLAI on the openchat_shareGPT4 dataset using the Mistral-7B model on an NVIDIA RTX ADA 6000 GPU. Compared to Sarathi-Serve, SLAI reduces the median TTFT by 53% and increases the maximum serving capacity by 26% such that median TTFT is below 0.5 seconds, while meeting tail TBT latency constraints. The complete source code is available at: https://github.com/agrimUT/SLAI.

Förderung

  • National Science Foundation

Zitieren

APA

Bari, A., Hegde, P., & Veciana, G. D. (2026). Optimal Scheduling Algorithms for LLM Inference: Theory and Practice. ACM SIGMETRICS Performance Evaluation Review, 54(1), 238-240. https://doi.org/10.1145/3820171.3806878

Vancouver

Bari A, Hegde P, Veciana GD. Optimal Scheduling Algorithms for LLM Inference: Theory and Practice. ACM SIGMETRICS Performance Evaluation Review. 2026;54(1):238-240. doi:10.1145/3820171.3806878

BibTeX
@article{Bari2026,
  author = {Bari, Agrim and Hegde, Parikshit and Veciana, Gustavo de},
  title = {Optimal Scheduling Algorithms for LLM Inference: Theory and Practice},
  journal = {ACM SIGMETRICS Performance Evaluation Review},
  year = {2026},
  volume = {54},
  number = {1},
  pages = {238--240},
  doi = {10.1145/3820171.3806878}
}
Kostenlos starten →