EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

CPU-GPU Workload Distribution during Throughput-Oriented LLM Inference on Single-GPU Systems

Daon Park, Bernhard Egger

ACM Transactions on Architecture and Code Optimization · 30.06.2026 · 23 (2) · 1-25

DOI: 10.1145/3816433 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, online erschienen am 25.06.2026. Verfasst von Daon Park und 1 weiteren Autor:innen. Die Arbeit nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Trade, Industry and Energy, Korean National Research Foundation, EU Horizon Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have recently achieved remarkable performance in text generation, capturing the attention of a broad audience. This success, driven by the rapid growth in model parameters, comes at the expense of significantly higher operational costs and decreased processing speed. These costs, combined with privacy concerns around cloud-based deployments, have motivated research into running LLMs on commodity hardware. For example, researchers have used the memory hierarchy to boost throughput by increasing the number of batches. These studies, however, tend to overlook or inefficiently utilize the additional computational resources provided by the CPU. In this work, we present a dynamic workload allocation technique that efficiently distributes computation across all available hardware resources. The proposed method targets decoder-based models on standard general-purpose hardware, effectively minimizing idle periods for both the CPU and the GPU. Experiments show that our approach achieves up to 30% higher throughput compared to the state of the art, regardless of model architecture, LLM optimizations, and input batch sizes.

Förderung

  • Ministry of Trade, Industry and Energy
  • Korean National Research Foundation
  • EU Horizon Program

Zitieren

APA

Park, D., & Egger, B. (2026). CPU-GPU Workload Distribution during Throughput-Oriented LLM Inference on Single-GPU Systems. ACM Transactions on Architecture and Code Optimization, 23(2), 1-25. https://doi.org/10.1145/3816433

Vancouver

Park D, Egger B. CPU-GPU Workload Distribution during Throughput-Oriented LLM Inference on Single-GPU Systems. ACM Transactions on Architecture and Code Optimization. 2026;23(2):1-25. doi:10.1145/3816433

BibTeX
@article{Park2026,
  author = {Park, Daon and Egger, Bernhard},
  title = {CPU-GPU Workload Distribution during Throughput-Oriented LLM Inference on Single-GPU Systems},
  journal = {ACM Transactions on Architecture and Code Optimization},
  year = {2026},
  volume = {23},
  number = {2},
  pages = {1--25},
  doi = {10.1145/3816433}
}
Kostenlos starten →