EUVIMEDEUVIMED entdecken
Crossref

TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics

Honghao Gao, Binyang Qiu, Ye Wang, Si Yu, Yueshen Xu, Xinheng Wang

IEEE Transactions on Consumer Electronics · 2024-02 · 70 (1) · 1134-1144

DOI: 10.1109/tce.2023.3339633 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Consumer Electronics, 2024-02. Verfasst von Honghao Gao und 5 weiteren Autor:innen. Die Arbeit wurde bisher 46-mal zitiert (Crossref) und nennt 39 Literaturangaben. Gefördert von: National Key Research and Development Program of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Consumer electronics such as mobile phones, wearable devices, and vehicle electronics use many intelligent applications such as voice commands, machine translation, and face recognition. These applications require large inference workloads to perform intelligent tasks, which are often completed using deep neural network (DNN) models. Traditional approaches rely on pure cloud computing, with consumer devices collecting data and cloud computing platforms completing inference tasks. In real life, the workloads of these applications are not fixed and are likely to exhibit fluctuations or unexpected surges, increasing the workload of cloud computing platforms. Simply increasing server resources often leads to resource waste. Therefore, a dynamic resource scheduling method is needed. In this paper, a token bucket-based dynamic batching (TBDB) algorithm that maintains throughput while reducing latency and increasing device utilization, especially for large volumes of requests, is proposed. Our work includes the following achievements: 1) We employ the token bucket algorithm to determine the workload, considering the concurrency and frequency of the data. We dynamically vary the maximum batch size (MBS) that will trigger the inference process for the next batch. 2) A low-coupling mode architecture that can be embedded into various consumer electronics in a plug-and-play manner is designed. 3) The performance of the electronic devices and the maximum latency are studied to provide guidance for setting hyperparameters. Finally, we evaluate the effectiveness of our method in three consumer electronic scenarios and present a theoretical analysis for setting hyperparameters in different scenarios.

Förderung

  • National Key Research and Development Program of China

Zitieren

APA

Gao, H., Qiu, B., Wang, Y., Yu, S., Xu, Y., & Wang, X. (2024). TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics. IEEE Transactions on Consumer Electronics, 70(1), 1134-1144. https://doi.org/10.1109/tce.2023.3339633

Vancouver

Gao H, Qiu B, Wang Y, Yu S, Xu Y, Wang X. TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics. IEEE Transactions on Consumer Electronics. 2024;70(1):1134-1144. doi:10.1109/tce.2023.3339633

BibTeX
@article{Gao2024,
  author = {Gao, Honghao and Qiu, Binyang and Wang, Ye and Yu, Si and Xu, Yueshen and Wang, Xinheng},
  title = {TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics},
  journal = {IEEE Transactions on Consumer Electronics},
  year = {2024},
  volume = {70},
  number = {1},
  pages = {1134--1144},
  doi = {10.1109/tce.2023.3339633}
}
Kostenlos starten →