EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Hermes: Efficient Serving of LLM Applications with Probabilistic Demand Modeling

Yifei Liu, Zuo Gan, Zhenghao Gan, Weiye Wang, Chen Chen, Yizhou Shan, Xusheng Chen, Zhenhua Han, Yifei Zhu, Shixuan Sun, Minyi Guo

ACM Transactions on Architecture and Code Optimization · 30.06.2026 · 23 (2) · 1-25

DOI: 10.1145/3803390 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, online erschienen am 27.06.2026. Verfasst von Yifei Liu und 10 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 88 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Applications based on Large Language Models (LLMs) contain a series of tasks to address real-world problems with boosted capability, which have dynamic demand volumes on diverse backends. Existing serving systems treat the resource demands of LLM applications as a blackbox, compromising end-to-end efficiency due to improper queuing order and backend warm up latency. We find that the resource demands of LLM applications can be modeled in a general and accurate manner with Probabilistic Demand Graph (PDGraph). We then propose Hermes, which leverages PDGraph for efficient serving of LLM applications. Confronting probabilistic demand description, Hermes applies the Gittins policy to determine the scheduling order that can minimize the average application completion time. It also uses the PDGraph model to help prewarm cold backends at proper moments. Experiments with diverse LLM applications confirm that Hermes can effectively improve the application serving efficiency, reducing the average completion time by over 70% and the P95 completion time by over 80%.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Liu, Y., Gan, Z., Gan, Z., Wang, W., Chen, C., Shan, Y., Chen, X., Han, Z., Zhu, Y., Sun, S., & Guo, M. (2026). Hermes: Efficient Serving of LLM Applications with Probabilistic Demand Modeling. ACM Transactions on Architecture and Code Optimization, 23(2), 1-25. https://doi.org/10.1145/3803390

Vancouver

Liu Y, Gan Z, Gan Z, Wang W, Chen C, Shan Y, et al. Hermes: Efficient Serving of LLM Applications with Probabilistic Demand Modeling. ACM Transactions on Architecture and Code Optimization. 2026;23(2):1-25. doi:10.1145/3803390

BibTeX
@article{Liu2026,
  author = {Liu, Yifei and Gan, Zuo and Gan, Zhenghao and Wang, Weiye and Chen, Chen and Shan, Yizhou and Chen, Xusheng and Han, Zhenhua and Zhu, Yifei and Sun, Shixuan and Guo, Minyi},
  title = {Hermes: Efficient Serving of LLM Applications with Probabilistic Demand Modeling},
  journal = {ACM Transactions on Architecture and Code Optimization},
  year = {2026},
  volume = {23},
  number = {2},
  pages = {1--25},
  doi = {10.1145/3803390}
}
Kostenlos starten →