mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs
Proceedings of the VLDB Endowment · 2025-02 · 18 (6) · 1948-1961
DOI: 10.14778/3725688.3725718 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 29.08.2025. Verfasst von Zhengmao Ye und 10 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 79 Literaturangaben.
Abstract (Originalsprache)
Transformer-based large language models (LLMs) have demonstrated outstanding performance across diverse domains, particularly in the emerging pretrain-then-finetune paradigm. LoRA, a parameter-efficient fine-tuning method, is commonly used to adapt a base LLM to multiple downstream tasks. Further, LLM platforms enable developers to fine-tune multiple models and develop various domain-specific applications simultaneously. However, existing model parallelism schemes suffer from high communication overhead and inefficient GPU utilization. In this paper, we present mLoRA, a parallelism-efficient fine-tuning system designed for training multiple LoRA across GPUs and machines. mLoRA introduces a novel LoRA-aware pipeline parallelism scheme that efficiently pipelines LoRA adapters and their distinct fine-tuning stages across GPUs and machines, along with a new LoRA-efficient operator to enhance GPU utilization. Our extensive evaluation shows that mLoRA can significantly reduce average fine-tuning task completion time, e.g., by 30%, compared to state-of-the-art methods like FSDP. More importantly, mLoRA enables simultaneous fine-tuning of larger models, e.g., two Llama-2-13B models on four NVIDIA RTX A6000 48GB GPUs, which is not feasible for FSDP due to high memory requirements. Hence, mLoRA not only increases fine-tuning efficiency but also makes it more accessible on cost-effective GPUs.
Zitieren
APA
Ye, Z., Li, D., Hu, Z., Lan, T., Sha, J., Zhang, S., Duan, L., Zuo, J., Lu, H., Zhou, Y., & Tang, M. (2025). mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs. Proceedings of the VLDB Endowment, 18(6), 1948-1961. https://doi.org/10.14778/3725688.3725718
Vancouver
Ye Z, Li D, Hu Z, Lan T, Sha J, Zhang S, et al. mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs. Proceedings of the VLDB Endowment. 2025;18(6):1948-1961. doi:10.14778/3725688.3725718
BibTeX
@article{Ye2025,
author = {Ye, Zhengmao and Li, Dengchun and Hu, Zetao and Lan, Tingfeng and Sha, Jian and Zhang, Shicong and Duan, Lei and Zuo, Jie and Lu, Hui and Zhou, Yuanchun and Tang, Mingjie},
title = {mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs},
journal = {Proceedings of the VLDB Endowment},
year = {2025},
volume = {18},
number = {6},
pages = {1948--1961},
doi = {10.14778/3725688.3725718}
}
EUVIMED