Seamless data delivery for distributed AI workloads via dynamic queue scheduling and FPGA-based implementation in converged optical-electrical networks
Journal of Optical Communications and Networking · 01.09.2025 · 17 (9) · 820
DOI: 10.1364/jocn.563049 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Optical Communications and Networking, online erschienen am 22.08.2025. Verfasst von Shi Feng und 5 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 43 Literaturangaben. Gefördert von: Huawei Technologies, National Natural Science Foundation of China.
Abstract (Originalsprache)
The deployment of universal AI training jobs in high-performance computing centers poses significant challenges to network architectures. Networks designed for artificial intelligence (AI) have emerged as a prevalent trend in datacenter networks. Traditional electrical packet-switching (EPS) networks face capacity limits due to the slowdown of Moore’s law and struggle to accommodate the specific traffic patterns of distributed deep learning (DDL). In contrast, optical circuit-switching (OCS) technology provides high-bandwidth, dedicated optical paths. The converged optical/electrical datacenter network (COE-DCN) has emerged as a promising solution for AI-DCN. However, optical circuit reconfiguration and multiplexing often introduce delays, disrupting traffic flow. Prior work in COE-DCN designs schedules the network, neglecting the negative influence during the optical path configuration. This paper addresses these challenges by analyzing traditional optical path provisioning methods and traffic-overlapping scenarios in COE-DCNs. We propose a bisection-assisted control mechanism to collaborate with optical and electrical networks, ensuring continuous data transmission. Our approach integrates queue-aware scheduling to dynamically allocate resources across EPS and OCS, minimizing transition latency and optimizing traffic flow. To validate the proposed scheme, we implement a field-programmable gate array (FPGA)-based hardware platform, which achieves sub-microsecond packet-level transition latency and demonstrates efficient queue management. Experimental results confirm significant improvements in job acceleration for overlapping DDL traffic scenarios, highlighting the effectiveness of our FPGA-based, queue-optimized design.
Förderung
- Huawei Technologies
- National Natural Science Foundation of China
Zitieren
APA
Feng, S., Zhang, J., Dai, J., Liu, Y., Liu, Z., & Ji, Y. (2025). Seamless data delivery for distributed AI workloads via dynamic queue scheduling and FPGA-based implementation in converged optical-electrical networks. Journal of Optical Communications and Networking, 17(9), 820. https://doi.org/10.1364/jocn.563049
Vancouver
Feng S, Zhang J, Dai J, Liu Y, Liu Z, Ji Y. Seamless data delivery for distributed AI workloads via dynamic queue scheduling and FPGA-based implementation in converged optical-electrical networks. Journal of Optical Communications and Networking. 2025;17(9):820. doi:10.1364/jocn.563049
BibTeX
@article{Feng2025,
author = {Feng, Shi and Zhang, Jiawei and Dai, Jun and Liu, Yashe and Liu, Zhenhua and Ji, Yuefeng},
title = {Seamless data delivery for distributed AI workloads via dynamic queue scheduling and FPGA-based implementation in converged optical-electrical networks},
journal = {Journal of Optical Communications and Networking},
year = {2025},
volume = {17},
number = {9},
pages = {820},
doi = {10.1364/jocn.563049}
}
EUVIMED