EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Online KV-Cache Aware Decoding Policies for Memory-Limited Large-Model Inference Deployments

Alex Yu, Henry So

International Journal of Business and Social Science Studies · 19.09.2026 · 6 (2) · 238-247

DOI: 10.66795/ijbsss75 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Business and Social Science Studies, online erschienen am 19.09.2026. Verfasst von Alex Yu und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The deployment of large language models in resource-constrained environments presents unprecedented challenges for memory management and computational efficiency. A primary bottleneck during the inference phase of these models is the key-value cache, which stores intermediate attention representations to prevent redundant computations. As sequence lengths and batch sizes increase, the memory footprint of the key-value cache grows linearly, frequently exceeding the physical memory capacities of edge devices and constrained server environments. This paper proposes a novel framework centered on online key-value cache aware decoding policies, which dynamically adjust inference parameters and memory allocation strategies in real time based on current memory pressure. By continuously monitoring the available high-bandwidth memory and dynamically modulating decoding strategies, such as adaptive sequence truncation, selective token eviction, and context-aware recomputation, the proposed framework maintains high generation quality while strictly adhering to hardware limitations. Extensive theoretical analysis and simulated deployment scenarios demonstrate that these online policies significantly reduce out-of-memory errors and improve overall system throughput compared to traditional static caching methodologies. The findings emphasize the necessity of transitioning from rigid memory allocation models to fluid, state-aware decoding algorithms. This research provides a foundational roadmap for optimizing large model deployments across diverse, memory-limited infrastructures, ultimately democratizing access to advanced artificial intelligence capabilities.

Zitieren

APA

Yu, A., & So, H. (2026). Online KV-Cache Aware Decoding Policies for Memory-Limited Large-Model Inference Deployments. International Journal of Business and Social Science Studies, 6(2), 238-247. https://doi.org/10.66795/ijbsss75

Vancouver

Yu A, So H. Online KV-Cache Aware Decoding Policies for Memory-Limited Large-Model Inference Deployments. International Journal of Business and Social Science Studies. 2026;6(2):238-247. doi:10.66795/ijbsss75

BibTeX
@article{Yu2026,
  author = {Yu, Alex and So, Henry},
  title = {Online KV-Cache Aware Decoding Policies for Memory-Limited Large-Model Inference Deployments},
  journal = {International Journal of Business and Social Science Studies},
  year = {2026},
  volume = {6},
  number = {2},
  pages = {238--247},
  doi = {10.66795/ijbsss75}
}
Kostenlos starten →