High-Ratio Compression for Machine-Generated Data
Proceedings of the ACM on Management of Data · 08.12.2023 · 1 (4) · 1-27
DOI: 10.1145/3626732 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Management of Data, online erschienen am 12.12.2023. Verfasst von Jiujing Zhang und 9 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: JSPS Kakenhi, National Key R&D Program of China, ARC Future Fellowship.
Abstract (Originalsprache)
Machine-generated data is rapidly growing and poses challenges for data-intensive systems, especially as the growth of data outpaces the growth of storage space. To cope with the storage issue, compression plays a critical role in storage engines, particularly for data-intensive applications, where a high compression ratio and efficient random access are essential. However, existing compression techniques tend to focus on general-purpose and data block approaches, but overlook the inherent structure of machine-generated data and hence result in low compression ratios or limited lookup efficiency. To address these limitations, we introduce the Pattern-Based Compression (PBC) algorithm, which specifically targets patterns in machine-generated data to achieve Pareto-optimality in most cases. Unlike traditional data block-based methods, PBC compresses data on a per-record basis, facilitating rapid random access. Our experimental evaluation demonstrates that PBC, on average, achieves a compression ratio twice as high as the state-of-the-art techniques while maintaining competitive compression and decompression speeds. We also integrate PBC to a production database system and achieve improvements on both comparison ratio and throughput.
Förderung
- JSPS Kakenhi
- National Key R&D Program of China
- ARC Future Fellowship
- NSFC
- the scholarship of China Scholarship Council
- CCF-AFSG Research Fund
- ARC Discovery Project
- CREST
Zitieren
APA
Zhang, J., Shen, Z., Yang, S., Meng, L., Xiao, C., Jia, W., Li, Y., Sun, Q., Zhang, W., & Lin, X. (2023). High-Ratio Compression for Machine-Generated Data. Proceedings of the ACM on Management of Data, 1(4), 1-27. https://doi.org/10.1145/3626732
Vancouver
Zhang J, Shen Z, Yang S, Meng L, Xiao C, Jia W, et al. High-Ratio Compression for Machine-Generated Data. Proceedings of the ACM on Management of Data. 2023;1(4):1-27. doi:10.1145/3626732
BibTeX
@article{Zhang2023,
author = {Zhang, Jiujing and Shen, Zhitao and Yang, Shiyu and Meng, Lingkai and Xiao, Chuan and Jia, Wei and Li, Yue and Sun, Qinhui and Zhang, Wenjie and Lin, Xuemin},
title = {High-Ratio Compression for Machine-Generated Data},
journal = {Proceedings of the ACM on Management of Data},
year = {2023},
volume = {1},
number = {4},
pages = {1--27},
doi = {10.1145/3626732}
}
EUVIMED