EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Characterizing KV-Cache Performance in Large Language Model Inference: Context Length, Attention Architecture, and GPU Memory Residency

Abed Al Raoof K. Bsoul, Yazan Alshboul

Journal of Low Power Electronics and Applications · 03.11.2026 · 16 (4) · 52

DOI: 10.3390/jlpea16040052 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Low Power Electronics and Applications, online erschienen am 03.11.2026. Verfasst von Abed Al Raoof K. Bsoul und 1 weiteren Autor:innen. Die Arbeit nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) commonly employ key–value (KV) caching to avoid recomputing previous attention states during autoregressive inference. However, its measured performance benefit depends on workload characteristics and the memory environment. This study presents a controlled systems characterization across input context length, model-specific grouped-query attention configurations, and GPU-memory residency. Experiments use Qwen2.5-3B and SmolLM3-3B under verified fully resident conditions with fixed 128-token generation, prompt lengths from 128 to 2048 tokens, randomized cache-enabled/cache-disabled execution, GPU telemetry, and repeated paired validation. End-to-end generation speedup increases from 1.34× to 8.31× for Qwen2.5-3B and from 1.28× to 8.62× for SmolLM3-3B. At 2048 prompt tokens, 30 randomized paired trials confirm mean speedups of 8.10× and 8.51×, corresponding to runtime reductions of 87.65% and 88.25%, while physical GPU-memory utilization remains below 61%. SmolLM3-3B requires twice the theoretical KV-cache storage per token, yet both models show comparable context-dependent scaling in the fully resident regime. Because the models differ in characteristics beyond KV-head count, this cross-model comparison is descriptive and does not attribute the observed performance difference causally to GQA configuration. A separate Mistral-7B case study shows that apparent end-to-end speedup changes substantially under memory pressure; the underlying memory-system mechanism is not isolated. These results establish cache-related end-to-end generation speedup as workload- and system-dependent and demonstrate that meaningful benchmarking requires reporting context length, cache configuration, model and cache footprints, physical accelerator-memory residency, runtime, throughput, and relevant GPU-memory conditions during each inference experiment.

Zitieren

APA

Bsoul, A. A. R. K., & Alshboul, Y. (2026). Characterizing KV-Cache Performance in Large Language Model Inference: Context Length, Attention Architecture, and GPU Memory Residency. Journal of Low Power Electronics and Applications, 16(4), 52. https://doi.org/10.3390/jlpea16040052

Vancouver

Bsoul AARK, Alshboul Y. Characterizing KV-Cache Performance in Large Language Model Inference: Context Length, Attention Architecture, and GPU Memory Residency. Journal of Low Power Electronics and Applications. 2026;16(4):52. doi:10.3390/jlpea16040052

BibTeX
@article{Bsoul2026,
  author = {Bsoul, Abed Al Raoof K. and Alshboul, Yazan},
  title = {Characterizing KV-Cache Performance in Large Language Model Inference: Context Length, Attention Architecture, and GPU Memory Residency},
  journal = {Journal of Low Power Electronics and Applications},
  year = {2026},
  volume = {16},
  number = {4},
  pages = {52},
  doi = {10.3390/jlpea16040052}
}
Kostenlos starten →