EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

OPTIMIZING DISTRIBUTED MACHINE LEARNING TRAINING: A COMPREHENSIVE REVIEW OF PARAMETER SERVER ARCHITECTURES, DATA PARTITIONING STRATEGIES, AND CONVERGENCE DYNAMICS

Arslan Ali, Muhammad Danish, Mobeen Mazhar, Maryam Siddique

Journal of Innovative Research and Technology · 2026-07-25 · 1 (1) · 1-10

DOI: 10.66857/b57a ↗

At a glance

Journal article in Journal of Innovative Research and Technology, published online on 2026-07-25. Written by Arslan Ali and 3 co-authors. The full text is freely available (open access, CC BY 4.0).

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

The increasing computational demands of modern machine learning have accelerated the adoption of distributed training frameworks capable of processing large-scale datasets and complex deep learning models. Among these frameworks, the Parameter Server (PS) architecture has become one of the most widely used approaches because of its scalability, flexibility, and compatibility with heterogeneous computing environments. Despite its widespread adoption, optimizing distributed training remains a challenging task due to the intricate relationships among system architecture, data partitioning mechanisms, communication efficiency, and convergence behaviors. This review presents a comprehensive analysis of recent advances in parameter server-based distributed machine learning by examining three interconnected research dimensions: parameter server architectures, data partitioning strategies, and convergence dynamics. The paper critically evaluates centralized, decentralized, and hierarchical parameter server designs, highlighting their respective strengths, limitations, and suitability for different distributed computing environments. Furthermore, it reviews widely adopted data partitioning techniques, including uniform random partitioning, feature-based partitioning, hybrid approaches, and non-independent and identically distributed (non-IID) data allocation, emphasizing their influence on workload balancing, communication overhead, and model performance. The review also investigates synchronization mechanisms such as synchronous, asynchronous, and bounded-staleness training, discussing their effects on gradient consistency, convergence stability, and computational efficiency. In addition, recent communication optimization techniques, including gradient compression, sparsification, and adaptive synchronization, are examined to illustrate their role in improving scalability while reducing network overhead. By synthesizing findings from contemporary studies, this paper identifies existing research gaps and highlights the complex interdependencies among architectural design, data distribution, and optimization strategies. The review concludes by outlining promising future research directions, including adaptive parameter server frameworks, intelligent partitioning methods, communication-aware optimization, and distributed learning systems designed for emerging large-scale artificial intelligence applications.

Cite

APA

Ali, A., Danish, M., Mazhar, M., & Siddique, M. (2026). OPTIMIZING DISTRIBUTED MACHINE LEARNING TRAINING: A COMPREHENSIVE REVIEW OF PARAMETER SERVER ARCHITECTURES, DATA PARTITIONING STRATEGIES, AND CONVERGENCE DYNAMICS. Journal of Innovative Research and Technology, 1(1), 1-10. https://doi.org/10.66857/b57a

Vancouver

Ali A, Danish M, Mazhar M, Siddique M. OPTIMIZING DISTRIBUTED MACHINE LEARNING TRAINING: A COMPREHENSIVE REVIEW OF PARAMETER SERVER ARCHITECTURES, DATA PARTITIONING STRATEGIES, AND CONVERGENCE DYNAMICS. Journal of Innovative Research and Technology. 2026;1(1):1-10. doi:10.66857/b57a

BibTeX
@article{Ali2026,
  author = {Ali, Arslan and Danish, Muhammad and Mazhar, Mobeen and Siddique, Maryam},
  title = {OPTIMIZING DISTRIBUTED MACHINE LEARNING TRAINING: A COMPREHENSIVE REVIEW OF PARAMETER SERVER ARCHITECTURES, DATA PARTITIONING STRATEGIES, AND CONVERGENCE DYNAMICS},
  journal = {Journal of Innovative Research and Technology},
  year = {2026},
  volume = {1},
  number = {1},
  pages = {1--10},
  doi = {10.66857/b57a}
}
Start for free →