EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Deduplication can Make File Systems Faster

Yanqi Pan, Wen Xia, Erci Xu, Hao Huang, Xiangyu Zou, Shiyi Li

ACM Transactions on Storage · 30.11.2026 · 22 (4) · 1-31

DOI: 10.1145/3806054 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Storage, online erschienen am 18.08.2026. Verfasst von Yanqi Pan und 5 weiteren Autor:innen. Die Arbeit nennt 69 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Major Key Project of PCL, National Natural Science Foundation of China, GuangDong Basic and Applied Basic Research Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Emerging storage technologies, such as persistent memory and ultra-low latency SSD, enable the deduplication file system (DedupFS) to use non-cryptographic hash for fast fingerprinting. However, we find that the accelerated computation exposes another major performance penalty: the seemingly innocuous in-storage deduplication metadata maintenance incurs up to 38% overhead in the I/O path. We find the root cause is that deduplication incurs additional crash consistency overhead due to the maintenance of dedup-specific fingerprint-to-physical mappings. However, our insight, called deduplication-assisted crash consistency , shows that the deduplication mapping does not necessarily incur overhead, but rather can be exploited to improve performance. Our insight is based on two key observations: (1) The deduplication mapping can be merged with the file system logical-to-physical mapping, forming a logical-fingerprint-physical (LFP) mapping. Thus, we can persist deduplication metadata alongside file system metadata in a single I/O; (2) Furthermore, this mapping, containing fingerprint of the data block, can be leveraged to verify data integrity. Thus, we can achieve orderless data writes by verifying the integrity during recovery. We propose DarthFS to realize the efficiency of deduplication-assisted crash consistency. Using a series of techniques to manage data, metadata, and orderless I/O, DarthFS successfully leverages deduplication to improve file system I/O performance while retaining compatibility, effectiveness, and memory efficiency. Experiments across a range of synthetic and real-world workloads show that DarthFS outperforms existing file systems even under the workloads without duplicates, consistently outperforms existing DedupFSes, and can minimize deduplication-induced crash consistency overhead.

Förderung

  • Major Key Project of PCL
  • National Natural Science Foundation of China
  • GuangDong Basic and Applied Basic Research Foundation

Zitieren

APA

Pan, Y., Xia, W., Xu, E., Huang, H., Zou, X., & Li, S. (2026). Deduplication can Make File Systems Faster. ACM Transactions on Storage, 22(4), 1-31. https://doi.org/10.1145/3806054

Vancouver

Pan Y, Xia W, Xu E, Huang H, Zou X, Li S. Deduplication can Make File Systems Faster. ACM Transactions on Storage. 2026;22(4):1-31. doi:10.1145/3806054

BibTeX
@article{Pan2026,
  author = {Pan, Yanqi and Xia, Wen and Xu, Erci and Huang, Hao and Zou, Xiangyu and Li, Shiyi},
  title = {Deduplication can Make File Systems Faster},
  journal = {ACM Transactions on Storage},
  year = {2026},
  volume = {22},
  number = {4},
  pages = {1--31},
  doi = {10.1145/3806054}
}
Kostenlos starten →