High-Bandwidth Flash for KV Caches: Endurance and Performance Implications
IEEE Computer Architecture Letters · 2026-01 · 25 (1) · 210-213
DOI: 10.1109/lca.2026.3695938 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Computer Architecture Letters, 2026-01. Verfasst von Kwanhee Kyung und 3 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 14 Literaturangaben.
Abstract (Originalsprache)
During Large Language Model (LLM) inference, the key-value (KV) cache is a critical bottleneck in both capacity and memory bandwidth as context lengths surge. While High-Bandwidth Memory (HBM) can meet current bandwidth demands, its capacity has not kept pace with the rapidly growing footprint of the KV cache. Consequently, KV caches are frequently offloaded to CPU memory for higher capacity, yet performance remains constrained by limited interconnect bandwidth. High-Bandwidth Flash (HBF) has emerged as a promising alternative, offering sequential read bandwidth comparable to HBM4 with up to an order-of-magnitude higher capacity. Although Flash memory has limited write endurance, KV caches exhibit a favorable access pattern, being written once and read many times during inference. Our analysis demonstrates that the KV cache read-to-write ratio increases with longer context lengths, particularly with larger output sequences. This characteristic enables KV caches to be stored in HBF while maintaining an acceptable device lifetime, addressing both the capacity and bandwidth challenges of contemporary LLM systems.
Zitieren
APA
Kyung, K., Moon, Y. J., Cho, J., & Ahn, J. H. (2026). High-Bandwidth Flash for KV Caches: Endurance and Performance Implications. IEEE Computer Architecture Letters, 25(1), 210-213. https://doi.org/10.1109/lca.2026.3695938
Vancouver
Kyung K, Moon YJ, Cho J, Ahn JH. High-Bandwidth Flash for KV Caches: Endurance and Performance Implications. IEEE Computer Architecture Letters. 2026;25(1):210-213. doi:10.1109/lca.2026.3695938
BibTeX
@article{Kyung2026,
author = {Kyung, Kwanhee and Moon, Yeon Ji and Cho, Juhwan and Ahn, Jung Ho},
title = {High-Bandwidth Flash for KV Caches: Endurance and Performance Implications},
journal = {IEEE Computer Architecture Letters},
year = {2026},
volume = {25},
number = {1},
pages = {210--213},
doi = {10.1109/lca.2026.3695938}
}
EUVIMED