Predicting Runtime in Spark‐Like Systems With Allocation‐Aware Deterministic Models
Software: Practice and Experience · 2026-12 · 56 (12) · 1449-1467
DOI: 10.1002/spe.70097 ↗Kurz gefasst
Fachartikel in der Zeitschrift Software: Practice and Experience, online erschienen am 04.08.2026. Verfasst von Hina Tariq und 1 weiteren Autor:innen. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Government of Canada.
Abstract (Originalsprache)
Objectives
Apache Spark is widely used for distributed data processing, but accurately predicting application execution time remains challenging because performance depends on application structure, resource configuration, and executor‐allocation behavior. This article presents two deterministic, graph‐based simulation models for predicting the execution time of Spark applications under static and dynamic executor allocation.
Methods
The first model, the Static Allocation Model (SAM), represents an application as a monolithic directed acyclic graph (DAG) of stages and simulates execution using Spark's scheduling behavior under a fixed number of executor cores. To capture more realistic execution environments, this framework is extended to a Dynamic Allocation Model (DAM), which accounts for runtime changes in executor resources using backlog‐time and idle‐time thresholds. DAM dynamically adjusts executor allocation during execution and integrates these changes within the deterministic scheduling framework.
Result
Validation of SAM on three TPC‐DS benchmark queries—Query‐64, Query‐26, and Query‐52—yielded prediction errors of 2.9%, 4.3%, and 0.4%, respectively, with a mean absolute error of 2.5%. Experimental evaluation of DAM on TPC‐DS Query‐52 and the SparkBench K‐Means workload yielded prediction errors of 5% and 4.7%, respectively.
Conclusion
To the best of our knowledge, this work presents the first deterministic approach for modeling Spark application execution time under dynamic executor allocation. The proposed models are lightweight, interpretable, and require minimal input data, offering a practical alternative for performance prediction and resource optimization in distributed data‐processing systems.
Förderung
- Government of Canada
Zitieren
APA
Tariq, H., & Das, O. (2026). Predicting Runtime in Spark‐Like Systems With Allocation‐Aware Deterministic Models. Software: Practice and Experience, 56(12), 1449-1467. https://doi.org/10.1002/spe.70097
Vancouver
Tariq H, Das O. Predicting Runtime in Spark‐Like Systems With Allocation‐Aware Deterministic Models. Software: Practice and Experience. 2026;56(12):1449-1467. doi:10.1002/spe.70097
BibTeX
@article{Tariq2026,
author = {Tariq, Hina and Das, Olivia},
title = {Predicting Runtime in Spark‐Like Systems With Allocation‐Aware Deterministic Models},
journal = {Software: Practice and Experience},
year = {2026},
volume = {56},
number = {12},
pages = {1449--1467},
doi = {10.1002/spe.70097}
}
EUVIMED