Distributed Machine Learning with Spark MLlib: A Comprehensive Performance Evaluation
Pakistan Journal of Scientific Research · 30.12.2025 · 5 (02) · 74-86
DOI: 10.57041/ya5rmz93 ↗Kurz gefasst
Fachartikel in der Zeitschrift Pakistan Journal of Scientific Research, online erschienen am 30.12.2025. Verfasst von Kiran Hayat und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).
Abstract (Originalsprache)
The growing demand for big data processing and distributed machine learning is necessary to apply predictive models to millions, or even billions, of records. Apache Spark MLlib is one of the most well-known Spark libraries, providing machine learning algorithms for processing large-scale data both in memory and in a distributed manner. Researchers have explored the distributed and in-memory processing capabilities of the Spark framework, little research has been conducted regarding empirical comparison of predictive performance and system efficiency using the Spark framework. These pipelines, classified as classification, regression, and clustering, are implemented in Spark MLlib. The experimental datasets consist of 540,000 to 12 M records with unified feature encoding and normalisation, and partitioning techniques. The evaluation metrics of the developed models include F1-Score, accuracy, Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Silhouette Score, Time spent on training, Memory Usage, and Computational Speedup. The study results show that the ensemble models (Random Forest and Gradient-Boosted Trees) outperform the others in classification. Linear Regression has lower regression errors, K-Means clustering is stable across different data sizes, and distributed computation speeds up computation while also providing better accuracy when applied to more complex models. These results provide insights into design decisions for both algorithms and partitioning strategies that can improve performance and accuracy in the production of Spark MLlib workflows.
Zitieren
APA
Hayat, K., Fatima, N., & Nazir, S. (2025). Distributed Machine Learning with Spark MLlib: A Comprehensive Performance Evaluation. Pakistan Journal of Scientific Research, 5(02), 74-86. https://doi.org/10.57041/ya5rmz93
Vancouver
Hayat K, Fatima N, Nazir S. Distributed Machine Learning with Spark MLlib: A Comprehensive Performance Evaluation. Pakistan Journal of Scientific Research. 2025;5(02):74-86. doi:10.57041/ya5rmz93
BibTeX
@article{Hayat2025,
author = {Hayat, Kiran and Fatima, Nargis and Nazir, Sumaira},
title = {Distributed Machine Learning with Spark MLlib: A Comprehensive Performance Evaluation},
journal = {Pakistan Journal of Scientific Research},
year = {2025},
volume = {5},
number = {02},
pages = {74--86},
doi = {10.57041/ya5rmz93}
}
EUVIMED