EUVIMEDEUVIMED entdecken
Crossref

Queueing, Predictions, and Large Language Models: Challenges and Open Problems

Michael Mitzenmacher, Rana Shahout

Stochastic Systems · 2025-09 · 15 (3) · 195-219

DOI: 10.1287/stsy.2025.0106 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Stochastic Systems, 2025-09. Verfasst von Michael Mitzenmacher und 1 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 36 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Queueing systems present many opportunities for applying machine learning predictions, such as estimated service times, to improve system performance. This integration raises numerous open questions about how predictions can be effectively leveraged to improve scheduling decisions. Recent studies explore queues with predicted service times, typically aiming to minimize job time in the system. We review these works, highlight the effectiveness of predictions, and present open questions on queue performance. We then move to consider an important practical example of using predictions in scheduling, namely large language model (LLM) systems, which presents novel scheduling challenges and highlights the potential for predictions to improve performance. In particular, we consider LLMs performing inference. Inference requests (jobs) in LLM systems are inherently complex; they have variable inference times, dynamic memory footprints that are constrained by key-value store memory limitations, and multiple possible preemption approaches that affect performance differently. We provide background on the important aspects of scheduling in LLM systems and introduce new models and open problems that arise from them. We argue that there are significant opportunities for applying insights and analysis from queueing theory to scheduling in LLM systems. Funding: M. Mitzenmacher was supported in part by the National Science Foundation [Grant CCF-2101140]. R. Shahout was supported in part by the Schmidt Futures Initiative and the Zuckerman Institute. M. Mitzenmacher and R. Shahout were supported in part the National Science Foundation [Grants CNS-2107078 and DMS-2023528].

Zitieren

APA

Mitzenmacher, M., & Shahout, R. (2025). Queueing, Predictions, and Large Language Models: Challenges and Open Problems. Stochastic Systems, 15(3), 195-219. https://doi.org/10.1287/stsy.2025.0106

Vancouver

Mitzenmacher M, Shahout R. Queueing, Predictions, and Large Language Models: Challenges and Open Problems. Stochastic Systems. 2025;15(3):195-219. doi:10.1287/stsy.2025.0106

BibTeX
@article{Mitzenmacher2025,
  author = {Mitzenmacher, Michael and Shahout, Rana},
  title = {Queueing, Predictions, and Large Language Models: Challenges and Open Problems},
  journal = {Stochastic Systems},
  year = {2025},
  volume = {15},
  number = {3},
  pages = {195--219},
  doi = {10.1287/stsy.2025.0106}
}
Kostenlos starten →