Automated Backend Allocation for Multi-Model, On-Device AI Inference
Proceedings of the ACM on Measurement and Analysis of Computing Systems · 07.12.2023 · 7 (3) · 1-33
DOI: 10.1145/3626793 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Measurement and Analysis of Computing Systems, online erschienen am 12.12.2023. Verfasst von Venkatraman Iyer und 5 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 97 Literaturangaben.
Abstract (Originalsprache)
On-Device Artificial Intelligence (AI) services such as face recognition, object tracking and voice recognition are rapidly scaling up deployments on embedded, memory-constrained hardware devices. These services typically delegate AI inference models for execution on CPU and GPU computing backends. While GPU delegation is a common practice to achieve high speed computation, the approach suffers from degraded throughput and completion times under multi-model scenarios, i.e. concurrently executing services. This paper introduces a solution to sustain performance in multi-model, on-device AI contexts by dynamically allocating a combination of CPU and GPU backends per model. The allocation is feedback-driven, and guided by a knowledge of model-specific, multi-objective pareto fronts comprising inference latency and memory consumption. Primary contribution of this paper is a backend allocation algorithm that runs online per model, and achieves 25-100% improvement in throughput over static allocations as well as load-balancing scheduler solutions targeting multi-model scenarios. Other noteworthy contributions include a novel pareto front estimator that runs on-device, and also a software-based GPU profiler with a lightweight algorithm to detect changing GPU workloads. Specifically, the pareto front estimator outperforms state of the art algorithms NSGA-II and SPEA2 by 94% on pareto coverage, and by almost 2x on computational overhead.
Zitieren
APA
Iyer, V., Lee, S., Lee, S., Kim, J. J., Kim, H., & Shin, Y. (2023). Automated Backend Allocation for Multi-Model, On-Device AI Inference. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 7(3), 1-33. https://doi.org/10.1145/3626793
Vancouver
Iyer V, Lee S, Lee S, Kim JJ, Kim H, Shin Y. Automated Backend Allocation for Multi-Model, On-Device AI Inference. Proceedings of the ACM on Measurement and Analysis of Computing Systems. 2023;7(3):1-33. doi:10.1145/3626793
BibTeX
@article{Iyer2023,
author = {Iyer, Venkatraman and Lee, Sungho and Lee, Semun and Kim, Juitem Joonwoo and Kim, Hyunjun and Shin, Youngjae},
title = {Automated Backend Allocation for Multi-Model, On-Device AI Inference},
journal = {Proceedings of the ACM on Measurement and Analysis of Computing Systems},
year = {2023},
volume = {7},
number = {3},
pages = {1--33},
doi = {10.1145/3626793}
}
EUVIMED