LIBPipe: Efficient Load Imbalance Pipeline Model Parallelism for Large Models Training
IEEE Transactions on Parallel and Distributed Systems · 2026-07 · 37 (7) · 1471-1484
DOI: 10.1109/tpds.2026.3686206 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Parallel and Distributed Systems, 2026-07. Verfasst von Bin Liu und 5 weiteren Autor:innen. Die Arbeit nennt 31 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Shaanxi’s Key Research and Development Program, Xianyang’s Key Research and Development Program.
Abstract (Originalsprache)
With the increasing size of datasets and the expansion of Deep Neural Networks (DNNs), the training process has become exceedingly time-consuming. Distributed training, specifically the Pipeline Model Parallelism (PMP) method, commonly mitigates this problem but suffers from bubble time delays. This paper proposes LIBPipe, a pipeline training framework that explicitly incorporates a load-imbalance method to reduce bubble time in PMP. Within LIBPipe, a model-unequal-partitioning method is designed from the perspective of load imbalance to reshape the pipeline execution pattern and significantly shorten idle periods during training. On top of this method, a performance-guided unequal-partitioning search algorithm is developed to efficiently identify near-optimal partitioning strategies under memory constraints. The paper theoretically proves the time efficiency of adopting load imbalance in pipeline models. Comprehensive experiments are conducted on an 8-GPU server to evaluate the efficiency of LIBPipe, using the IMDB and mini-ImageNet datasets as well as well-known models such as BERT and ResNet. The BERT-series models achieve a maximum throughput improvement of 60.3%, while the ResNet-series models achieve a maximum improvement of 74.1%.
Förderung
- National Natural Science Foundation of China
- Shaanxi’s Key Research and Development Program
- Xianyang’s Key Research and Development Program
- Natural Science Basic Research Program of Shaanxi Province
Zitieren
APA
Liu, B., Zhang, Z., Li, H., Ji, Z., Zhang, H., & Li, K. (2026). LIBPipe: Efficient Load Imbalance Pipeline Model Parallelism for Large Models Training. IEEE Transactions on Parallel and Distributed Systems, 37(7), 1471-1484. https://doi.org/10.1109/tpds.2026.3686206
Vancouver
Liu B, Zhang Z, Li H, Ji Z, Zhang H, Li K. LIBPipe: Efficient Load Imbalance Pipeline Model Parallelism for Large Models Training. IEEE Transactions on Parallel and Distributed Systems. 2026;37(7):1471-1484. doi:10.1109/tpds.2026.3686206
BibTeX
@article{Liu2026,
author = {Liu, Bin and Zhang, Zhonghao and Li, Hengzhao and Ji, Zeyu and Zhang, Hongming and Li, Keqin},
title = {LIBPipe: Efficient Load Imbalance Pipeline Model Parallelism for Large Models Training},
journal = {IEEE Transactions on Parallel and Distributed Systems},
year = {2026},
volume = {37},
number = {7},
pages = {1471--1484},
doi = {10.1109/tpds.2026.3686206}
}
EUVIMED