PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING
Innovation and Technology Studies · 2025-12-18 · 2 (2) · 38-44
DOI: 10.61784/its03024 ↗At a glance
Journal article in Innovation and Technology Studies, published online on 2025-12-18. Written by Dai Teng. The full text is freely available (open access, CC BY 4.0).
Abstract (original language)
Serving large language models (LLMs) is expensive, and production workloads fluctuate by an order of magnitude within hours while requests must meet tight latency service-level objectives (SLOs) on time-to-first-token (TTFT) and time-per-output-token (TPOT). Operators today either provision statically for peak load, wasting most of the fleet off-peak, or rely on reactive autoscalers that reconfigure only after violations are observed. This paper presents PACO, a predictive auto-configuration framework that periodically selects the cheapest serving configuration expected to satisfy tail-latency SLOs. PACO combines (i) a lightweight workload forecaster, (ii) a gradient-boosted performance model that predicts windowed P95 TTFT/TPOT from workload features and a candidate configuration, learned from offline profiling, (iii) an SLO-constrained minimum-cost search over the joint space of replica count and chunked-prefill token budget, and (iv) a feedback guard that bounds the impact of prediction errors. We evaluate PACO with an iteration-level cluster simulator driven by production LLM inference traces released by Microsoft Azure. On a diurnal conversation workload, PACO reduces SLO-violating windows from 27.6% to 3.4% at 2% lower cost than a reactive autoscaler, cuts cost by 60.5% relative to static peak provisioning, and stays within 27% of a clairvoyant oracle’s cost. Ablations show the forecaster halves violations and the joint tuning of the token budget is essential for long-context workloads.
Cite
APA
Teng, D. (2025). PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING. Innovation and Technology Studies, 2(2), 38-44. https://doi.org/10.61784/its03024
Vancouver
Teng D. PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING. Innovation and Technology Studies. 2025;2(2):38-44. doi:10.61784/its03024
BibTeX
@article{Teng2025,
author = {Teng, Dai},
title = {PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING},
journal = {Innovation and Technology Studies},
year = {2025},
volume = {2},
number = {2},
pages = {38--44},
doi = {10.61784/its03024}
}
EUVIMED