EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference

Juncheol Ahn iD, Yubin Son, Daemin Kim, Sejin Park iD

Sensors · 08.02.2026 · 26 (4) · 1101

DOI: 10.3390/s26041101 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 08.02.2026. Verfasst von Juncheol Ahn und 3 weiteren Autor:innen. Beteiligte Einrichtung: System Software Laboratory, Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Daegu Digital Innovation Promotion Agency(DIP), Daegu Digital Innovation Promotion Agency(DIP) OF FUNDER.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models in IoT–edge–cloud settings face bursty, heterogeneous requests that make pipeline-parallel inference prone to micro-batch imbalance and communication stalls, causing GPU idle time and SLO violations. We propose a runtime-adaptive scheduler that jointly tunes token budgets and micro-batch counts to balance prefill/decode workloads and minimize pipeline bubbles under changing compute and network conditions. On a four-node pipeline-parallel cluster across Llama-2-13b and Qwen2.5-14b at 100/1000 Mbps, our method outperforms vLLM and SGLang, reducing GPU idle time by up to 55% and improving throughput by up to 1.61 × while improving TTFT/ITL SLO satisfaction. These results show that dynamic scheduling is essential for scalable, latency-stable LLM inference in IoT–edge–cloud environments.

Schlagwörter

Stichwörter der Autor:innen

GPU schedulingIoTcloud inferenceedge computinglarge language modelsmicro-batchingpipeline parallelism

Autor:innen und Institutionen

  • System Software Laboratory, Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea.

ORCID-Profile: Juncheol Ahn, Sejin Park

Förderung

  • Daegu Digital Innovation Promotion Agency(DIP)
  • Daegu Digital Innovation Promotion Agency(DIP) OF FUNDER

Zitieren

APA

Ahn, J., Son, Y., Kim, D., & Park, S. (2026). Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference. Sensors, 26(4), 1101. https://doi.org/10.3390/s26041101

Vancouver

Ahn J, Son Y, Kim D, Park S. Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference. Sensors. 2026;26(4):1101. doi:10.3390/s26041101

BibTeX
@article{Ahn2026,
  author = {Ahn, Juncheol and Son, Yubin and Kim, Daemin and Park, Sejin},
  title = {Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference},
  journal = {Sensors},
  year = {2026},
  volume = {26},
  number = {4},
  pages = {1101},
  doi = {10.3390/s26041101}
}
Kostenlos starten →