EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Optimization of the Join between Large Tables in the Spark Distributed Framework

Xiang Wu, Yueshun He

Applied Sciences · 19.05.2023 · 13 (10) · 6257

DOI: 10.3390/app13106257 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 19.05.2023. Verfasst von Xiang Wu und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Key Research Projects of Jiangxi Province, Jiangxi Key Laboratory of Cybersecurity Intelligent Perception.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Join task between Spark large tables takes a long time to run and produces a lot of disk I/O, network I/O and disk occupation in the Shuffle process. This paper proposes a lightweight distributed data filtering model that combines broadcast variables and accumulators using RoaringBitmap. When the data in the two tables are not exactly matched, the dimension table Key is collected through the accumulator, compressed by RoaringBitmap and distributed to each node using broadcast variables. The distributed fact table data can be pre-filtered on the local server, which effectively reduces the data transmission and disk reading and writing in the Shuffle phase. Experimental results show that this optimization method can reduce disk usage, shorten the running time and reduce network I/O and disk I/O for Spark Join tasks in the case of massive data, and the effect is more obvious when the two tables have a higher incomplete matching degree or a fixed matching degree but a larger amount of data. This optimization scheme has the advantages of being easy to use, being easy to maintain and having an obvious effect, and it can be applied to many development scenarios.

Förderung

  • Key Research Projects of Jiangxi Province
  • Jiangxi Key Laboratory of Cybersecurity Intelligent Perception

Zitieren

APA

Wu, X., & He, Y. (2023). Optimization of the Join between Large Tables in the Spark Distributed Framework. Applied Sciences, 13(10), 6257. https://doi.org/10.3390/app13106257

Vancouver

Wu X, He Y. Optimization of the Join between Large Tables in the Spark Distributed Framework. Applied Sciences. 2023;13(10):6257. doi:10.3390/app13106257

BibTeX
@article{Wu2023,
  author = {Wu, Xiang and He, Yueshun},
  title = {Optimization of the Join between Large Tables in the Spark Distributed Framework},
  journal = {Applied Sciences},
  year = {2023},
  volume = {13},
  number = {10},
  pages = {6257},
  doi = {10.3390/app13106257}
}
Kostenlos starten →