EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency

Sihyeong Park, Sungryeol Jeon, Chaelyn Lee, Seokhun Jeon, Byung-Soo Kim, Jemin Lee

ACM Transactions on Intelligent Systems and Technology · 28.02.2027 · 18 (1) · 1-111

DOI: 10.1145/3803798 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 29.10.2026. Verfasst von Sihyeong Park und 5 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 408 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Institute of Information and Communications Technology Planning and Evaluation, Jeonbuk National University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) are widely applied in chatbots, code generators, and search engines. Workload such as chain-of-thought, complex reasoning, and agent services significantly increase inference cost by invoking the model repeatedly. Optimization methods such as parallelism, compression, and caching have been adopted to reduce costs, but the diverse service requirements make it hard to select the right method. Recently, specialized LLM inference engines have emerged as a key component for integrating the optimization methods into service-oriented infrastructures. However, a systematic study on inference engines is still lacking. This article provides a comprehensive evaluation of 25 open source and commercial inference engines. We analyze each inference engine in terms of ease-of-use, ease-of-deployment, general-purpose support, scalability, and suitability for throughput- and latency-aware computation. Furthermore, we explore the design goals of each inference engine by investigating the optimization techniques it supports. In addition, we assess the ecosystem maturity of open source inference engines and handle the performance and cost policy of commercial solutions. We outline future research directions that include support for complex LLM-based services, support of various hardware, and enhanced security, offering practical guidance to researchers and developers in selecting and designing optimized LLM inference engines. We also provide a public repository to continually track developments in this fast-evolving field: https://github.com/sihyeong/Awesome-LLM-Inference-Engine .

Förderung

  • Institute of Information and Communications Technology Planning and Evaluation
  • Jeonbuk National University

Zitieren

APA

Park, S., Jeon, S., Lee, C., Jeon, S., Kim, B., & Lee, J. (2027). A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency. ACM Transactions on Intelligent Systems and Technology, 18(1), 1-111. https://doi.org/10.1145/3803798

Vancouver

Park S, Jeon S, Lee C, Jeon S, Kim B, Lee J. A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency. ACM Transactions on Intelligent Systems and Technology. 2027;18(1):1-111. doi:10.1145/3803798

BibTeX
@article{Park2027,
  author = {Park, Sihyeong and Jeon, Sungryeol and Lee, Chaelyn and Jeon, Seokhun and Kim, Byung-Soo and Lee, Jemin},
  title = {A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2027},
  volume = {18},
  number = {1},
  pages = {1--111},
  doi = {10.1145/3803798}
}
Kostenlos starten →