EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Capacity Signals for Model Serving: Engine Telemetry Versus Device Metrics

Kailash Mannepalli

Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 · 29.09.2026 · 10 (01) · 219-223

DOI: 10.60087/jaigs.v10i01.514 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, online erschienen am 29.09.2026. Verfasst von Kailash Mannepalli. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Device-level GPU telemetry, such as NVIDIA DCGM framebuffer and utilization counters, is the default input to Kubernetes autoscalers, but it misreports capacity for large language model (LLM) serving. Paged-KV runtimes such as vLLM reserve a fixed fraction of framebuffer at startup, so DCGM_FI_DEV_FB_USED is constant from idle to overload, and DCGM_FI_DEV_GPU_UTIL reports kernel residency rather than queued work. We measure this divergence on two NVIDIA A100-80GB GPUs serving Llama-3-8B-Instruct under closed-loop load with mixed context lengths. Across five load phases from 0 to 250 concurrent requests, framebuffer usage stayed at 91.8% throughout, while engine-side KV-cache occupancy rose from 0% to 100%, waiting-queue depth from 0 to 86, and preemption rate from 0 to 14.8 per second. Compute utilization separated the phases below saturation but moved only from 88.6% to 99.2% between 60 and 250 concurrent requests, the range in which queueing and preemption appeared. We argue that KV-cache occupancy, queue depth and preemption rate are the correct scaling signals, show that occupancy-aware routing of long-context requests lowers p99 time-to-first-token by 32% near saturation, and describe local weight caching to cut scale-out cold start. The setup uses only vLLM, DCGM-Exporter, Prometheus and a small routing proxy, and reproduces in about an hour of GPU time.

Zitieren

APA

Mannepalli, K. (2026). Capacity Signals for Model Serving: Engine Telemetry Versus Device Metrics. Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, 10(01), 219-223. https://doi.org/10.60087/jaigs.v10i01.514

Vancouver

Mannepalli K. Capacity Signals for Model Serving: Engine Telemetry Versus Device Metrics. Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023. 2026;10(01):219-223. doi:10.60087/jaigs.v10i01.514

BibTeX
@article{Mannepalli2026,
  author = {Mannepalli, Kailash},
  title = {Capacity Signals for Model Serving: Engine Telemetry Versus Device Metrics},
  journal = {Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023},
  year = {2026},
  volume = {10},
  number = {01},
  pages = {219--223},
  doi = {10.60087/jaigs.v10i01.514}
}
Kostenlos starten →