Data-Aware Adaptive Compression for Stream Processing
IEEE Transactions on Knowledge and Data Engineering · 2024-09 · 36 (9) · 4531-4549
DOI: 10.1109/tkde.2024.3377710 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Knowledge and Data Engineering, 2024-09. Verfasst von Yu Zhang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 91 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Beijing Nova Program.
Abstract (Originalsprache)
Stream processing has been in widespread use, and one of the most common application scenarios is SQL query on streams. By 2021, the global deployment of IoT endpoints reached 12.3 billion, indicating a surge in data generation. However, the escalating demands for high throughput and low latency in stream processing systems have posed significant challenges due to the increasing data volume and evolving user requirements. We present a compression-based stream processing engine, called CompressStreamDB, which enables adaptive fine-grained stream processing directly on compressed streams, to significantly enhance the performance of existing stream processing solutions. CompressStreamDB utilizes nine diverse compression methods tailored for different stream data types and integrates a cost model to automatically select the most efficient compression schemes. CompressStreamDB provides high throughput with low latency in stream SQL processing by identifying and eliminating redundant data among streams. Our evaluation demonstrates that CompressStreamDB improves average performance by 3.84× and reduces average delay by 68.0% compared to the state-of-the-art stream processing solution for uncompressed streams, along with 68.7% space savings. Besides, our edge trials show an average throughput/price ratio of 9.95× and a throughput/power ratio of 7.32× compared to the cloud design.
Förderung
- National Natural Science Foundation of China
- Beijing Nova Program
Zitieren
APA
Zhang, Y., Zhang, F., Li, H., Zhang, S., Guo, X., Chen, Y., Pan, A., & Du, X. (2024). Data-Aware Adaptive Compression for Stream Processing. IEEE Transactions on Knowledge and Data Engineering, 36(9), 4531-4549. https://doi.org/10.1109/tkde.2024.3377710
Vancouver
Zhang Y, Zhang F, Li H, Zhang S, Guo X, Chen Y, et al. Data-Aware Adaptive Compression for Stream Processing. IEEE Transactions on Knowledge and Data Engineering. 2024;36(9):4531-4549. doi:10.1109/tkde.2024.3377710
BibTeX
@article{Zhang2024,
author = {Zhang, Yu and Zhang, Feng and Li, Hourun and Zhang, Shuhao and Guo, Xiaoguang and Chen, Yuxing and Pan, Anqun and Du, Xiaoyong},
title = {Data-Aware Adaptive Compression for Stream Processing},
journal = {IEEE Transactions on Knowledge and Data Engineering},
year = {2024},
volume = {36},
number = {9},
pages = {4531--4549},
doi = {10.1109/tkde.2024.3377710}
}
EUVIMED