Online KV-Cache Aware Decoding Policies for Memory-Limited Large-Model Inference Deployments
International Journal of Business and Social Science Studies · 19.09.2026 · 6 (2) · 238-247
DOI: 10.66795/ijbsss75 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Business and Social Science Studies, online erschienen am 19.09.2026. Verfasst von Alex Yu und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The deployment of large language models in resource-constrained environments presents unprecedented challenges for memory management and computational efficiency. A primary bottleneck during the inference phase of these models is the key-value cache, which stores intermediate attention representations to prevent redundant computations. As sequence lengths and batch sizes increase, the memory footprint of the key-value cache grows linearly, frequently exceeding the physical memory capacities of edge devices and constrained server environments. This paper proposes a novel framework centered on online key-value cache aware decoding policies, which dynamically adjust inference parameters and memory allocation strategies in real time based on current memory pressure. By continuously monitoring the available high-bandwidth memory and dynamically modulating decoding strategies, such as adaptive sequence truncation, selective token eviction, and context-aware recomputation, the proposed framework maintains high generation quality while strictly adhering to hardware limitations. Extensive theoretical analysis and simulated deployment scenarios demonstrate that these online policies significantly reduce out-of-memory errors and improve overall system throughput compared to traditional static caching methodologies. The findings emphasize the necessity of transitioning from rigid memory allocation models to fluid, state-aware decoding algorithms. This research provides a foundational roadmap for optimizing large model deployments across diverse, memory-limited infrastructures, ultimately democratizing access to advanced artificial intelligence capabilities.
Zitieren
APA
Yu, A., & So, H. (2026). Online KV-Cache Aware Decoding Policies for Memory-Limited Large-Model Inference Deployments. International Journal of Business and Social Science Studies, 6(2), 238-247. https://doi.org/10.66795/ijbsss75
Vancouver
Yu A, So H. Online KV-Cache Aware Decoding Policies for Memory-Limited Large-Model Inference Deployments. International Journal of Business and Social Science Studies. 2026;6(2):238-247. doi:10.66795/ijbsss75
BibTeX
@article{Yu2026,
author = {Yu, Alex and So, Henry},
title = {Online KV-Cache Aware Decoding Policies for Memory-Limited Large-Model Inference Deployments},
journal = {International Journal of Business and Social Science Studies},
year = {2026},
volume = {6},
number = {2},
pages = {238--247},
doi = {10.66795/ijbsss75}
}
EUVIMED