FedFMSL: Federated Learning of Foundation Models With Sparsely Activated LoRA
IEEE Transactions on Mobile Computing · 2024-12 · 23 (12) · 15167-15181
DOI: 10.1109/tmc.2024.3454634 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2024-12. Verfasst von Panlong Wu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 37-mal zitiert (Crossref) und nennt 42 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Hetao Shenzhen-HK S&T Cooperation Zone, Shenzhen Science and Technology Program.
Abstract (Originalsprache)
Foundation models (FMs) have shown great success in natural language processing, computer vision, and multimodal tasks. FMs have a large number of model parameters, thus requiring a substantial amount of data to help optimize the model during the training. Federated learning has revolutionized machine learning by enabling collaborative learning from decentralized data while still preserving clients’ data privacy. Despite the great benefits foundation models can have empowered by federated learning, their bulky model parameters cause severe communication challenges for modern networks and computation challenges especially for edge devices. Moreover, the data distribution of different clients can be different thus inducing statistical challenges. In this paper, we propose a novel two-stage federated learning algorithm called FedFMSL. A global expert is trained in the first stage and a local expert is trained in the second stage to provide better personalization. We construct a Mixture of Foundation Models (MoFM) with these two experts and design a gate neural network with an inserted gate adapter that joins the aggregation every communication round in the second stage. To further adapt to edge computing scenarios with limited computational resources, we design a novel Sparsely Activated LoRA (SAL) algorithm that freezes the pre-trained foundation model parameters inserts low-rank adaptation matrices into transformer blocks, and activates them progressively during the training. We employ extensive experiments to verify the effectiveness of FedFMSL, results show that FedFMSL outperforms other SOTA baselines by up to 59.19% in default settings while tuning less than 0.3% parameters of the foundation model.
Förderung
- National Natural Science Foundation of China
- Hetao Shenzhen-HK S&T Cooperation Zone
- Shenzhen Science and Technology Program
- Basic and Applied Basic Research Foundation of Guangdong Province
- Shenzhen Outstanding Talents Training Fund
- Guangdong Research Projects
- Guangdong Provincial Key Laboratory of Future Networks of Intelligence
- Pearl River Talent Recruitment Program of Guangdong Province
Zitieren
APA
Wu, P., Li, K., Wang, T., Dong, Y., Leung, V. C. M., & Wang, F. (2024). FedFMSL: Federated Learning of Foundation Models With Sparsely Activated LoRA. IEEE Transactions on Mobile Computing, 23(12), 15167-15181. https://doi.org/10.1109/tmc.2024.3454634
Vancouver
Wu P, Li K, Wang T, Dong Y, Leung VCM, Wang F. FedFMSL: Federated Learning of Foundation Models With Sparsely Activated LoRA. IEEE Transactions on Mobile Computing. 2024;23(12):15167-15181. doi:10.1109/tmc.2024.3454634
BibTeX
@article{Wu2024,
author = {Wu, Panlong and Li, Kangshuo and Wang, Ting and Dong, Yanjie and Leung, Victor C. M. and Wang, Fangxin},
title = {FedFMSL: Federated Learning of Foundation Models With Sparsely Activated LoRA},
journal = {IEEE Transactions on Mobile Computing},
year = {2024},
volume = {23},
number = {12},
pages = {15167--15181},
doi = {10.1109/tmc.2024.3454634}
}
EUVIMED