FLCD: A Flexible Low Complexity Design of Coded Distributed Computing
IEEE Transactions on Cloud Computing · 01.01.2023 · 11 (1) · 470-483
DOI: 10.1109/tcc.2021.3098593 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Cloud Computing, erschienen am 01.01.2023. Verfasst von Nicholas Woolsey und 3 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 29 Literaturangaben. Gefördert von: National Science Foundation.
Abstract (Originalsprache)
We propose a flexible low complexity design (FLCD) of coded distributed computing (CDC) with empirical evaluation on Amazon Elastic Compute Cloud (Amazon EC2). CDC can expedite MapReduce like computation by trading increased map computations to reduce communication load and shuffle time. A main novelty of FLCD is to utilize the design freedom in defining map and reduce functions to develop asymptotic homogeneous systems to support varying intermediate values (IV) sizes under a general MapReduce framework. Compared to existing designs with constant IV sizes, FLCD offers greater flexibility in adapting to network parameters and significantly reduces the implementation complexity by requiring fewer input files and shuffle groups. The FLCD scheme is the first proposed low-complexity CDC design that can operate on a network with an arbitrary number of nodes and computation load. We perform empirical evaluations of the FLCD by executing the TeraSort algorithm on an Amazon EC2 cluster. This is the first time that theoretical predictions of the CDC shuffle time are validated by empirical evaluations. The evaluations demonstrate a 2.0 to 4.24× speedup compared to conventional uncoded MapReduce, a 12 to 52 percent reduction in total time, and a wider range of operating network parameters compared to existing CDC schemes.
Förderung
- National Science Foundation
Zitieren
APA
Woolsey, N., Wang, X., Chen, R., & Ji, M. (2023). FLCD: A Flexible Low Complexity Design of Coded Distributed Computing. IEEE Transactions on Cloud Computing, 11(1), 470-483. https://doi.org/10.1109/tcc.2021.3098593
Vancouver
Woolsey N, Wang X, Chen R, Ji M. FLCD: A Flexible Low Complexity Design of Coded Distributed Computing. IEEE Transactions on Cloud Computing. 2023;11(1):470-483. doi:10.1109/tcc.2021.3098593
BibTeX
@article{Woolsey2023,
author = {Woolsey, Nicholas and Wang, Xingyue and Chen, Rong-Rong and Ji, Mingyue},
title = {FLCD: A Flexible Low Complexity Design of Coded Distributed Computing},
journal = {IEEE Transactions on Cloud Computing},
year = {2023},
volume = {11},
number = {1},
pages = {470--483},
doi = {10.1109/tcc.2021.3098593}
}
EUVIMED