EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Inference-Time-Driven Autoscaling for Inference Workloads: A Comparative Study of Latency-Variant Models in Kubernetes

Josephine Eskaline Joyce, Shoney Sebastian

Technologies · 10.06.2026 · 14 (6) · 350

DOI: 10.3390/technologies14060350 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Technologies, online erschienen am 10.06.2026. Verfasst von Josephine Eskaline Joyce und 1 weiteren Autor:innen. Die Arbeit nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Kubernetes Horizontal Pod Autoscaler (HPA) primarily relies on resource-based metrics, such as CPU utilization, which are poorly suited to capturing the latency variability of AI inference workloads. In this paper, we propose a custom-metric-driven autoscaling approach that leverages inference latency histograms as first-class scaling signals for Kubernetes HPA. The proposed framework integrates a Prometheus Operator (PO)-based observability stack with the Prometheus Adapter to expose and aggregate per-pod inference latency metrics, enabling workload-aware scaling decisions. We evaluate the approach using four mid-scale transformer-based inference services, comprising two reasoning-like and two latency-stable workloads, under high-concurrency conditions. The experiments analyze latency variation, tail behavior, and replica dynamics across multiple autoscaling policies, including variations in scale-up aggressiveness (3 pods/30 s, 3 pods/60 s, 6 pods/60 s), inference-time thresholds, and stabilization windows. Compared to CPU-based autoscaling, inference-driven policies reduce mean response time by 18–27% for reasoning-like workloads and 12–20% for stable workloads. The results show that latency-variable workloads exhibit wider tails and higher variance, indicating the need for moderately aggressive scale-up strategies to avoid long-lasting latency spikes. Overall, the findings show that inference-latency-driven custom metrics significantly improve autoscaling efficiency and stability for transformer-based inference workloads in cloud-native environments.

Zitieren

APA

Joyce, J. E., & Sebastian, S. (2026). Inference-Time-Driven Autoscaling for Inference Workloads: A Comparative Study of Latency-Variant Models in Kubernetes. Technologies, 14(6), 350. https://doi.org/10.3390/technologies14060350

Vancouver

Joyce JE, Sebastian S. Inference-Time-Driven Autoscaling for Inference Workloads: A Comparative Study of Latency-Variant Models in Kubernetes. Technologies. 2026;14(6):350. doi:10.3390/technologies14060350

BibTeX
@article{Joyce2026,
  author = {Joyce, Josephine Eskaline and Sebastian, Shoney},
  title = {Inference-Time-Driven Autoscaling for Inference Workloads: A Comparative Study of Latency-Variant Models in Kubernetes},
  journal = {Technologies},
  year = {2026},
  volume = {14},
  number = {6},
  pages = {350},
  doi = {10.3390/technologies14060350}
}
Kostenlos starten →