EUVIMEDEUVIMED entdecken
Open Access · CC BY-SA 4.0

Distributed Machine Learning with Spark MLlib: A Comprehensive Performance Evaluation

Kiran Hayat, Nargis Fatima, Sumaira Nazir

Pakistan Journal of Scientific Research · 30.12.2025 · 5 (02) · 74-86

DOI: 10.57041/ya5rmz93 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Pakistan Journal of Scientific Research, online erschienen am 30.12.2025. Verfasst von Kiran Hayat und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The growing demand for big data processing and distributed machine learning is necessary to apply predictive models to millions, or even billions, of records. Apache Spark MLlib is one of the most well-known Spark libraries, providing machine learning algorithms for processing large-scale data both in memory and in a distributed manner. Researchers have explored the distributed and in-memory processing capabilities of the Spark framework, little research has been conducted regarding empirical comparison of predictive performance and system efficiency using the Spark framework. These pipelines, classified as classification, regression, and clustering, are implemented in Spark MLlib. The experimental datasets consist of 540,000 to 12 M records with unified feature encoding and normalisation, and partitioning techniques. The evaluation metrics of the developed models include F1-Score, accuracy, Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Silhouette Score, Time spent on training, Memory Usage, and Computational Speedup. The study results show that the ensemble models (Random Forest and Gradient-Boosted Trees) outperform the others in classification. Linear Regression has lower regression errors, K-Means clustering is stable across different data sizes, and distributed computation speeds up computation while also providing better accuracy when applied to more complex models. These results provide insights into design decisions for both algorithms and partitioning strategies that can improve performance and accuracy in the production of Spark MLlib workflows.

Zitieren

APA

Hayat, K., Fatima, N., & Nazir, S. (2025). Distributed Machine Learning with Spark MLlib: A Comprehensive Performance Evaluation. Pakistan Journal of Scientific Research, 5(02), 74-86. https://doi.org/10.57041/ya5rmz93

Vancouver

Hayat K, Fatima N, Nazir S. Distributed Machine Learning with Spark MLlib: A Comprehensive Performance Evaluation. Pakistan Journal of Scientific Research. 2025;5(02):74-86. doi:10.57041/ya5rmz93

BibTeX
@article{Hayat2025,
  author = {Hayat, Kiran and Fatima, Nargis and Nazir, Sumaira},
  title = {Distributed Machine Learning with Spark MLlib: A Comprehensive Performance Evaluation},
  journal = {Pakistan Journal of Scientific Research},
  year = {2025},
  volume = {5},
  number = {02},
  pages = {74--86},
  doi = {10.57041/ya5rmz93}
}
Kostenlos starten →