EUVIMEDEUVIMED entdecken
Crossref

LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure

Jaehong Cho, Hyunmin Choi, Jongse Park

IEEE Computer Architecture Letters · 2025-07 · 24 (2) · 361-364

DOI: 10.1109/lca.2025.3628325 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Computer Architecture Letters, 2025-07. Verfasst von Jaehong Cho und 2 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 12 Literaturangaben. Gefördert von: Institute of Information & Communications Technology Planning, National Research Foundation of Korea, Electronics and Telecommunications Research Institute.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This letter introduces LLMServingSim2.0, a system simulator designed for exploring heterogeneous hardware in large-scale LLM serving systems. LLMServingSim2.0 addresses two key limitations of its predecessor: (1) integrating hardware models into system-level simulators is non-trivial due to the lack of a clear abstraction, and (2) existing simulators support only a narrow subset of serving techniques, leaving no infrastructure that captures the breadth of approaches in modern LLM serving. To overcome these issues, LLMServingSim2.0 adopts trace-driven performance modeling, accompanied by an operator-level latency profiler, enabling the integration of new accelerators with a single command. It further embeds up-to-date serving techniques while exposing flexible interfaces for request routing, cache management, and scheduling policies. In a TPU case study, our profiler requires 18.5 × fewer LoC and outperforms the predecessor’s hardware-simulator integration, demonstrating LLMServingSim2.0’s low-effort hardware extensibility. Our experiments further show that LLMServingSim2.0 reproduces GPU-based LLM serving with 1.9% error, while maintaining practical simulation time, making it a comprehensive platform for both hardware developers and LLM service providers.

Förderung

  • Institute of Information & Communications Technology Planning
  • National Research Foundation of Korea
  • Electronics and Telecommunications Research Institute
  • Korea Government

Zitieren

APA

Cho, J., Choi, H., & Park, J. (2025). LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure. IEEE Computer Architecture Letters, 24(2), 361-364. https://doi.org/10.1109/lca.2025.3628325

Vancouver

Cho J, Choi H, Park J. LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure. IEEE Computer Architecture Letters. 2025;24(2):361-364. doi:10.1109/lca.2025.3628325

BibTeX
@article{Cho2025,
  author = {Cho, Jaehong and Choi, Hyunmin and Park, Jongse},
  title = {LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure},
  journal = {IEEE Computer Architecture Letters},
  year = {2025},
  volume = {24},
  number = {2},
  pages = {361--364},
  doi = {10.1109/lca.2025.3628325}
}
Kostenlos starten →