EUVIMEDEUVIMED entdecken
Crossref

Koala: Efficient Pipeline Training through Automated Schedule Searching on Domain-Specific Language

Yu Tang, Lujia Yin, Qiao Li, Hongyu Zhu, Hengjie Li, Xingcheng Zhang, Linbo Qiao, Dongsheng Li, Jiaxin Li

ACM Transactions on Architecture and Code Optimization · 30.06.2025 · 22 (2) · 1-25

DOI: 10.1145/3722113 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, 2025-07. Verfasst von Yu Tang und 8 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Pipeline parallelism is a crucial technique for large-scale model training, enabling parameter splitting and performance enhancement. However, creating effective pipeline schedules often requires significant manual effort and coding skills, leading to practical inconveniences and complex debugging. Major frameworks such as DeepSpeed and ColossalAI simplify the process by adopting predefined pipeline schedule strategies, such as GPipe and 1F1B. The use of predefined schedules offers limited flexibility and suboptimal training efficiency, as the limited number of manually set candidates cannot provide the optimal strategy for arbitrary model training. To deal with the issue, this article aims to automatically search for the optimal strategy with high efficiency. Since current frameworks only support a limited set of fixed strategies, lacking the technical capability to create a comprehensive strategy search space, we first design a novel domain-specific language (DSL) for pipeline schedule development. The DSL exhibits great understandability, agility, and reusability, supporting the development of all known pipeline schedule strategies and their variants. Second, we are the first to model the complete pipeline schedule strategy space via the DSL, enabling an automated end-to-end globally optimal pipeline schedule searching, while past work may get stuck in a local optimum. Finally, we propose to optimize pipeline performance by modeling and solving the pipeline schedule as a Binary-Tree-Traversing (BTT) optimization problem. Based on the formalization, we further adopt a Dynamic Try-Test Genetic Algorithm to search for the best pipeline schedule strategy, which overwhelms a variety of pre-defined ones. Experimental results show that Koala achieves an enhanced performance by up to \(1.53\times\) over state-of-the-art approaches. Besides, the pipeline schedule strategy searched by Koala outperforms pre-defined pipeline schedule strategies by \(1.10\times \sim 1.55\times\) . Moreover, Koala has superior scalability and effectiveness in combining with data parallelism and tensor parallelism.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Tang, Y., Yin, L., Li, Q., Zhu, H., Li, H., Zhang, X., Qiao, L., Li, D., & Li, J. (2025). Koala: Efficient Pipeline Training through Automated Schedule Searching on Domain-Specific Language. ACM Transactions on Architecture and Code Optimization, 22(2), 1-25. https://doi.org/10.1145/3722113

Vancouver

Tang Y, Yin L, Li Q, Zhu H, Li H, Zhang X, et al. Koala: Efficient Pipeline Training through Automated Schedule Searching on Domain-Specific Language. ACM Transactions on Architecture and Code Optimization. 2025;22(2):1-25. doi:10.1145/3722113

BibTeX
@article{Tang2025,
  author = {Tang, Yu and Yin, Lujia and Li, Qiao and Zhu, Hongyu and Li, Hengjie and Zhang, Xingcheng and Qiao, Linbo and Li, Dongsheng and Li, Jiaxin},
  title = {Koala: Efficient Pipeline Training through Automated Schedule Searching on Domain-Specific Language},
  journal = {ACM Transactions on Architecture and Code Optimization},
  year = {2025},
  volume = {22},
  number = {2},
  pages = {1--25},
  doi = {10.1145/3722113}
}
Kostenlos starten →