EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

EcoInfer: Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control in LLM Serving

Qingyuan Hu, Jian Li

Electronics · 16.05.2026 · 15 (10) · 2139

DOI: 10.3390/electronics15102139 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 16.05.2026. Verfasst von Qingyuan Hu und 1 weiteren Autor:innen. Die Arbeit nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language model (LLM) serving has emerged as a major source of energy consumption in modern AI infrastructure. In current deployments, graphics processing units (GPUs) are typically operated at default high-frequency settings to maximize performance. However, under practical service-level objectives (SLOs), peak performance is often unnecessary, especially during the memory-bound decode stage, resulting in substantial power redundancy and avoidable energy waste. Existing studies that apply GPU dynamic voltage and frequency scaling (DVFS) to improve the energy efficiency of LLM serving have shown promising results. However, they generally rely on coarse-grained control, accurate output length prediction, or request-level resource management, which limits their effectiveness under highly dynamic workloads and strict SLO constraints. We present EcoInfer, a fine-grained DVFS framework for energy-efficient LLM serving. EcoInfer performs iteration-level, workload-aware GPU frequency control that adapts to the current inference phase and system state while preserving latency guarantees. It comprises three tightly integrated modules: a machine-learning-based frequency–latency predictor that estimates iteration latency across candidate GPU frequencies using lightweight iteration-level features; an SLO-aware frequency controller that selects the minimum feasible frequency within a sweet-spot-guided candidate range; and a low-overhead runtime optimization layer that combines adaptive decision caching with asynchronous execution to reduce and hide the overhead of online control. Implemented on top of vLLM, EcoInfer achieves up to 25.4% energy savings and 21.5% average energy savings and improves energy efficiency by 1.28× on average in terms of Tokens/J while maintaining a nearly unchanged SLO attainment rate compared with the default vLLM baseline.

Förderung

  • Natural Science Foundation of China

Zitieren

APA

Hu, Q., & Li, J. (2026). EcoInfer: Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control in LLM Serving. Electronics, 15(10), 2139. https://doi.org/10.3390/electronics15102139

Vancouver

Hu Q, Li J. EcoInfer: Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control in LLM Serving. Electronics. 2026;15(10):2139. doi:10.3390/electronics15102139

BibTeX
@article{Hu2026,
  author = {Hu, Qingyuan and Li, Jian},
  title = {EcoInfer: Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control in LLM Serving},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {10},
  pages = {2139},
  doi = {10.3390/electronics15102139}
}
Kostenlos starten →