Adaptive Fragment-Based Parallel State Recovery for Stream Processing Systems
IEEE Transactions on Parallel and Distributed Systems · 2023-08 · 34 (8) · 2464-2478
DOI: 10.1109/tpds.2023.3251997 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Parallel and Distributed Systems, 2023-08. Verfasst von Hailu Xu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 82 Literaturangaben. Gefördert von: National Science Foundation.
Abstract (Originalsprache)
Today, large-scale cloud organizations are deploying datacenters and “edge” clusters globally to provide low-latency access to services. Running stream applications across geo-distributed sites are emerging as a daily requirement. However, existing efforts have dominantly centered around stateless stream processing, leaving another urgent trend-stateful stream processing-much less explored. A driving need is to store and update states during processing, and most importantly, successfully recover large distributed states when faults and failures happen. Existing studies exhibit major limitations including: (1) they mostly inherit MapReduce's “single master/many workers” architecture, where the central master can easily become ascalability bottleneck; (2) they offer state recovery mainly through three approaches: replication recovery, checkpointing recovery, and DStream-based lineage recovery, which are either slow, resource-expensive or failing to handle multiple failures; and (3) they are not adaptive to heterogeneous hardware settings. We present A-FP4S, a novel adaptive fragments-based parallel state recovery mechanism for stream processing systems. A-FP4S organizes stream operators into a distributed hash table based peer-to-peer overlay and divides each node's local state into many fragments. These fragments are periodically stored in node's multiple neighbors, ensuring different sets of available fragments can reconstruct failed states in parallel. This mechanism is extremely scalable to the lost state, significantly reduces failure recovery time, and can tolerate multiple node failures. A-FP4S is adaptive to heterogeneous hardware settings by automatic parameter tuning over phases. Compared to Apache Storm, A-FP4S achieves 31.8% to 50.5% reduction in recovery latency. Large-scale experiments using real-world datasets demonstrate A-FP4S's attractive scalability and adaptivity properties.
Förderung
- National Science Foundation
Zitieren
APA
Xu, H., Liu, P., Ahmed, S. T., Silva, D. D., & Hu, L. (2023). Adaptive Fragment-Based Parallel State Recovery for Stream Processing Systems. IEEE Transactions on Parallel and Distributed Systems, 34(8), 2464-2478. https://doi.org/10.1109/tpds.2023.3251997
Vancouver
Xu H, Liu P, Ahmed ST, Silva DD, Hu L. Adaptive Fragment-Based Parallel State Recovery for Stream Processing Systems. IEEE Transactions on Parallel and Distributed Systems. 2023;34(8):2464-2478. doi:10.1109/tpds.2023.3251997
BibTeX
@article{Xu2023,
author = {Xu, Hailu and Liu, Pinchao and Ahmed, Sarker Tanzir and Silva, Dilma Da and Hu, Liting},
title = {Adaptive Fragment-Based Parallel State Recovery for Stream Processing Systems},
journal = {IEEE Transactions on Parallel and Distributed Systems},
year = {2023},
volume = {34},
number = {8},
pages = {2464--2478},
doi = {10.1109/tpds.2023.3251997}
}
EUVIMED