EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Characterizing Directionally Asymmetric Job-Level Transfer Behavior in HTCondor Clusters: An Empirical Study of Pattern Discovery and Submission-Time Predictability

Mingshan Xia, Li Wang, Jiahong Xu, Fazhi Qi

Computers · 04.11.2026 · 15 (11) · 801

DOI: 10.3390/computers15110801 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers, online erschienen am 04.11.2026. Verfasst von Mingshan Xia und 3 weiteren Autor:innen. Die Arbeit nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, National High Energy Physics Science Data Center Greater Bay Area Branch.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The network fabric of geographically distributed high-throughput computing (HTC) clusters has become a critical factor in supporting large-scale scientific workflows, yet job-level data transfer behavior in production batch systems remains insufficiently understood. In this paper, we present a measurement and predictability study of HTCondor-recorded job-level transfer behavior using a 14-month production dataset comprising 757,868 real jobs from the geographically distributed HTCondor cluster at the Institute of High Energy Physics (IHEP). Our analysis reveals a pronounced directional asymmetry in recorded job-level transfer behavior when measured by job count: 88.4% of jobs exhibit only input-side transfer, while 92.9% generate zero output-side transfer. By applying unsupervised clustering as an exploratory labeling tool, we derive empirical labels for seven input-side transfer patterns and three output-side transfer patterns, revealing both dominant low-transfer populations and rare high-transfer groups. We further use lightweight job information available at submission time to examine whether these discovered transfer patterns can be predicted before execution. The results show that input-side transfer patterns are partially predictable from job semantics such as job type, accounting group, and command-line arguments, while output-side prediction is largely constrained by severe class imbalance and dominant job-type regularities. Overall, the results provide empirical evidence that job-level transfer behavior in a real geographically distributed HTC environment is structured, directionally asymmetric, and partially predictable before execution. These findings offer lightweight transfer-awareness signals for future studies of data-locality-aware site ranking, cache preparation, storage layout planning, and transfer-aware resource management in large-scale HTC systems.

Förderung

  • National Natural Science Foundation of China
  • National High Energy Physics Science Data Center Greater Bay Area Branch

Zitieren

APA

Xia, M., Wang, L., Xu, J., & Qi, F. (2026). Characterizing Directionally Asymmetric Job-Level Transfer Behavior in HTCondor Clusters: An Empirical Study of Pattern Discovery and Submission-Time Predictability. Computers, 15(11), 801. https://doi.org/10.3390/computers15110801

Vancouver

Xia M, Wang L, Xu J, Qi F. Characterizing Directionally Asymmetric Job-Level Transfer Behavior in HTCondor Clusters: An Empirical Study of Pattern Discovery and Submission-Time Predictability. Computers. 2026;15(11):801. doi:10.3390/computers15110801

BibTeX
@article{Xia2026,
  author = {Xia, Mingshan and Wang, Li and Xu, Jiahong and Qi, Fazhi},
  title = {Characterizing Directionally Asymmetric Job-Level Transfer Behavior in HTCondor Clusters: An Empirical Study of Pattern Discovery and Submission-Time Predictability},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {11},
  pages = {801},
  doi = {10.3390/computers15110801}
}
Kostenlos starten →