EUVIMEDEUVIMED entdecken
Crossref

High-Bandwidth Flash for KV Caches: Endurance and Performance Implications

Kwanhee Kyung, Yeon Ji Moon, Juhwan Cho, Jung Ho Ahn

IEEE Computer Architecture Letters · 2026-01 · 25 (1) · 210-213

DOI: 10.1109/lca.2026.3695938 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Computer Architecture Letters, 2026-01. Verfasst von Kwanhee Kyung und 3 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 14 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

During Large Language Model (LLM) inference, the key-value (KV) cache is a critical bottleneck in both capacity and memory bandwidth as context lengths surge. While High-Bandwidth Memory (HBM) can meet current bandwidth demands, its capacity has not kept pace with the rapidly growing footprint of the KV cache. Consequently, KV caches are frequently offloaded to CPU memory for higher capacity, yet performance remains constrained by limited interconnect bandwidth. High-Bandwidth Flash (HBF) has emerged as a promising alternative, offering sequential read bandwidth comparable to HBM4 with up to an order-of-magnitude higher capacity. Although Flash memory has limited write endurance, KV caches exhibit a favorable access pattern, being written once and read many times during inference. Our analysis demonstrates that the KV cache read-to-write ratio increases with longer context lengths, particularly with larger output sequences. This characteristic enables KV caches to be stored in HBF while maintaining an acceptable device lifetime, addressing both the capacity and bandwidth challenges of contemporary LLM systems.

Zitieren

APA

Kyung, K., Moon, Y. J., Cho, J., & Ahn, J. H. (2026). High-Bandwidth Flash for KV Caches: Endurance and Performance Implications. IEEE Computer Architecture Letters, 25(1), 210-213. https://doi.org/10.1109/lca.2026.3695938

Vancouver

Kyung K, Moon YJ, Cho J, Ahn JH. High-Bandwidth Flash for KV Caches: Endurance and Performance Implications. IEEE Computer Architecture Letters. 2026;25(1):210-213. doi:10.1109/lca.2026.3695938

BibTeX
@article{Kyung2026,
  author = {Kyung, Kwanhee and Moon, Yeon Ji and Cho, Juhwan and Ahn, Jung Ho},
  title = {High-Bandwidth Flash for KV Caches: Endurance and Performance Implications},
  journal = {IEEE Computer Architecture Letters},
  year = {2026},
  volume = {25},
  number = {1},
  pages = {210--213},
  doi = {10.1109/lca.2026.3695938}
}
Kostenlos starten →