EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING

Dai Teng

Innovation and Technology Studies · 2025-12-18 · 2 (2) · 38-44

DOI: 10.61784/its03024 ↗

At a glance

Journal article in Innovation and Technology Studies, published online on 2025-12-18. Written by Dai Teng. The full text is freely available (open access, CC BY 4.0).

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Serving large language models (LLMs) is expensive, and production workloads fluctuate by an order of magnitude within hours while requests must meet tight latency service-level objectives (SLOs) on time-to-first-token (TTFT) and time-per-output-token (TPOT). Operators today either provision statically for peak load, wasting most of the fleet off-peak, or rely on reactive autoscalers that reconfigure only after violations are observed. This paper presents PACO, a predictive auto-configuration framework that periodically selects the cheapest serving configuration expected to satisfy tail-latency SLOs. PACO combines (i) a lightweight workload forecaster, (ii) a gradient-boosted performance model that predicts windowed P95 TTFT/TPOT from workload features and a candidate configuration, learned from offline profiling, (iii) an SLO-constrained minimum-cost search over the joint space of replica count and chunked-prefill token budget, and (iv) a feedback guard that bounds the impact of prediction errors. We evaluate PACO with an iteration-level cluster simulator driven by production LLM inference traces released by Microsoft Azure. On a diurnal conversation workload, PACO reduces SLO-violating windows from 27.6% to 3.4% at 2% lower cost than a reactive autoscaler, cuts cost by 60.5% relative to static peak provisioning, and stays within 27% of a clairvoyant oracle’s cost. Ablations show the forecaster halves violations and the joint tuning of the token budget is essential for long-context workloads.

Cite

APA

Teng, D. (2025). PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING. Innovation and Technology Studies, 2(2), 38-44. https://doi.org/10.61784/its03024

Vancouver

Teng D. PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING. Innovation and Technology Studies. 2025;2(2):38-44. doi:10.61784/its03024

BibTeX
@article{Teng2025,
  author = {Teng, Dai},
  title = {PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING},
  journal = {Innovation and Technology Studies},
  year = {2025},
  volume = {2},
  number = {2},
  pages = {38--44},
  doi = {10.61784/its03024}
}
Start for free →