EUVIMEDEUVIMED entdecken
Crossref

The Design of Fast and Lightweight Resemblance Detection for Efficient Post-Deduplication Delta Compression

Wen Xia, Lifeng Pu, Xiangyu Zou, Philip Shilane, Shiyi Li, Haijun Zhang, Xuan Wang

ACM Transactions on Storage · 31.08.2023 · 19 (3) · 1-30

DOI: 10.1145/3584663 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Storage, online erschienen am 19.06.2023. Verfasst von Wen Xia und 6 weiteren Autor:innen. Die Arbeit wurde bisher 102-mal zitiert (Crossref) und nennt 44 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Guangdong Basic and Applied Basic Research Foundation, Shenzhen Science and Technology Innovation Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Post-deduplication delta compression is a data reduction technique that calculates and stores the differences of very similar but non-duplicate chunks in storage systems, which is able to achieve a very high compression ratio. However, the low throughput of widely used resemblance detection approaches (e.g., N-Transform) usually becomes the bottleneck of delta compression systems due to introducing high computational overhead. Generally, this overhead mainly consists of two parts: ① calculating the rolling hash byte by byte across data chunks and ② applying multiple transforms on all of the calculated rolling hash values. In this article, we propose Odess, a fast and lightweight resemblance detection approach, that greatly reduces the computational overhead for resemblance detection while achieving high detection accuracy and a high compression ratio. Odess first utilizes a novel Subwindow-based Parallel Rolling (SWPR) hash method using Single Instruction Multiple Data [ 1 ] (SIMD) to accelerate calculation of rolling hashes (corresponding to the first part of the overhead). Odess then uses a novel Content-Defined Sampling method to generate a much smaller proxy hash set from the whole rolling hash set and quickly applies transforms on this small hash set for resemblance detection (corresponding to the second part of the overhead). Evaluation results show that during the stage of resemblance detection, the Odess approach is ∼31.4× and ∼7.9× faster than the state-of-the-art N-Transform and Finesse (a recent variant of N-Transform [ 39 ]), respectively. When considering an end-to-end data reduction storage system, the Odess-based system’s throughput is about 3.20× and 1.41× higher than the N-Transform- and Finesse-based systems’ throughput, respectively, while maintaining the high compression ratio of N-Transform and achieving ∼1.22× higher compression ratio over Finesse.

Förderung

  • National Natural Science Foundation of China
  • Guangdong Basic and Applied Basic Research Foundation
  • Shenzhen Science and Technology Innovation Program
  • Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies
  • HITSZ-J&A Joint Laboratory of Digital Design and Intelligent Fabrication

Zitieren

APA

Xia, W., Pu, L., Zou, X., Shilane, P., Li, S., Zhang, H., & Wang, X. (2023). The Design of Fast and Lightweight Resemblance Detection for Efficient Post-Deduplication Delta Compression. ACM Transactions on Storage, 19(3), 1-30. https://doi.org/10.1145/3584663

Vancouver

Xia W, Pu L, Zou X, Shilane P, Li S, Zhang H, et al. The Design of Fast and Lightweight Resemblance Detection for Efficient Post-Deduplication Delta Compression. ACM Transactions on Storage. 2023;19(3):1-30. doi:10.1145/3584663

BibTeX
@article{Xia2023,
  author = {Xia, Wen and Pu, Lifeng and Zou, Xiangyu and Shilane, Philip and Li, Shiyi and Zhang, Haijun and Wang, Xuan},
  title = {The Design of Fast and Lightweight Resemblance Detection for Efficient Post-Deduplication Delta Compression},
  journal = {ACM Transactions on Storage},
  year = {2023},
  volume = {19},
  number = {3},
  pages = {1--30},
  doi = {10.1145/3584663}
}
Kostenlos starten →