Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference
Sensors · 08.02.2026 · 26 (4) · 1101
DOI: 10.3390/s26041101 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 08.02.2026. Verfasst von Juncheol Ahn und 3 weiteren Autor:innen. Beteiligte Einrichtung: System Software Laboratory, Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Daegu Digital Innovation Promotion Agency(DIP), Daegu Digital Innovation Promotion Agency(DIP) OF FUNDER.
Abstract (Originalsprache)
Large language models in IoT–edge–cloud settings face bursty, heterogeneous requests that make pipeline-parallel inference prone to micro-batch imbalance and communication stalls, causing GPU idle time and SLO violations. We propose a runtime-adaptive scheduler that jointly tunes token budgets and micro-batch counts to balance prefill/decode workloads and minimize pipeline bubbles under changing compute and network conditions. On a four-node pipeline-parallel cluster across Llama-2-13b and Qwen2.5-14b at 100/1000 Mbps, our method outperforms vLLM and SGLang, reducing GPU idle time by up to 55% and improving throughput by up to 1.61 × while improving TTFT/ITL SLO satisfaction. These results show that dynamic scheduling is essential for scalable, latency-stable LLM inference in IoT–edge–cloud environments.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- System Software Laboratory, Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea.
ORCID-Profile: Juncheol Ahn, Sejin Park
Förderung
- Daegu Digital Innovation Promotion Agency(DIP)
- Daegu Digital Innovation Promotion Agency(DIP) OF FUNDER
Zitieren
APA
Ahn, J., Son, Y., Kim, D., & Park, S. (2026). Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference. Sensors, 26(4), 1101. https://doi.org/10.3390/s26041101
Vancouver
Ahn J, Son Y, Kim D, Park S. Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference. Sensors. 2026;26(4):1101. doi:10.3390/s26041101
BibTeX
@article{Ahn2026,
author = {Ahn, Juncheol and Son, Yubin and Kim, Daemin and Park, Sejin},
title = {Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference},
journal = {Sensors},
year = {2026},
volume = {26},
number = {4},
pages = {1101},
doi = {10.3390/s26041101}
}
EUVIMED