EUVIMEDEUVIMED entdecken
Crossref

LIBPipe: Efficient Load Imbalance Pipeline Model Parallelism for Large Models Training

Bin Liu, Zhonghao Zhang, Hengzhao Li, Zeyu Ji, Hongming Zhang, Keqin Li

IEEE Transactions on Parallel and Distributed Systems · 2026-07 · 37 (7) · 1471-1484

DOI: 10.1109/tpds.2026.3686206 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Parallel and Distributed Systems, 2026-07. Verfasst von Bin Liu und 5 weiteren Autor:innen. Die Arbeit nennt 31 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Shaanxi’s Key Research and Development Program, Xianyang’s Key Research and Development Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the increasing size of datasets and the expansion of Deep Neural Networks (DNNs), the training process has become exceedingly time-consuming. Distributed training, specifically the Pipeline Model Parallelism (PMP) method, commonly mitigates this problem but suffers from bubble time delays. This paper proposes LIBPipe, a pipeline training framework that explicitly incorporates a load-imbalance method to reduce bubble time in PMP. Within LIBPipe, a model-unequal-partitioning method is designed from the perspective of load imbalance to reshape the pipeline execution pattern and significantly shorten idle periods during training. On top of this method, a performance-guided unequal-partitioning search algorithm is developed to efficiently identify near-optimal partitioning strategies under memory constraints. The paper theoretically proves the time efficiency of adopting load imbalance in pipeline models. Comprehensive experiments are conducted on an 8-GPU server to evaluate the efficiency of LIBPipe, using the IMDB and mini-ImageNet datasets as well as well-known models such as BERT and ResNet. The BERT-series models achieve a maximum throughput improvement of 60.3%, while the ResNet-series models achieve a maximum improvement of 74.1%.

Förderung

  • National Natural Science Foundation of China
  • Shaanxi’s Key Research and Development Program
  • Xianyang’s Key Research and Development Program
  • Natural Science Basic Research Program of Shaanxi Province

Zitieren

APA

Liu, B., Zhang, Z., Li, H., Ji, Z., Zhang, H., & Li, K. (2026). LIBPipe: Efficient Load Imbalance Pipeline Model Parallelism for Large Models Training. IEEE Transactions on Parallel and Distributed Systems, 37(7), 1471-1484. https://doi.org/10.1109/tpds.2026.3686206

Vancouver

Liu B, Zhang Z, Li H, Ji Z, Zhang H, Li K. LIBPipe: Efficient Load Imbalance Pipeline Model Parallelism for Large Models Training. IEEE Transactions on Parallel and Distributed Systems. 2026;37(7):1471-1484. doi:10.1109/tpds.2026.3686206

BibTeX
@article{Liu2026,
  author = {Liu, Bin and Zhang, Zhonghao and Li, Hengzhao and Ji, Zeyu and Zhang, Hongming and Li, Keqin},
  title = {LIBPipe: Efficient Load Imbalance Pipeline Model Parallelism for Large Models Training},
  journal = {IEEE Transactions on Parallel and Distributed Systems},
  year = {2026},
  volume = {37},
  number = {7},
  pages = {1471--1484},
  doi = {10.1109/tpds.2026.3686206}
}
Kostenlos starten →