Federated Low-Rank Adaptation for Large Models Fine-Tuning Over Wireless Networks
IEEE Transactions on Wireless Communications · 2025-01 · 24 (1) · 659-675
DOI: 10.1109/twc.2024.3497998 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Wireless Communications, 2025-01. Verfasst von Haofeng Sun und 5 weiteren Autor:innen. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 54 Literaturangaben.
Abstract (Originalsprache)
The emergence of large language models (LLMs) with multi-task generalization capabilities is expected to improve the performance of artificial intelligence (AI)-as-a-service provision in 6G networks. By fine-tuning LLMs, AI services can become more precise and tailored to the demands of different downstream tasks. However, centralized fine-tuning paradigms pose a potential risk to user privacy, and existing distributed fine-tuning methods incur significant wireless transmission burdens due to the large-scale parameter transmission of LLMs. To tackle these challenges, by leveraging the low rank feature in LLM fine-tuning, we propose a wireless over-the-air federated learning (AirFL) based low-rank adaptation (LoRA) framework that integrates LoRA and over-the-air computation (AirComp) to achieve efficient fine-tuning and aggregation. Based on multiple-input multiple-output (MIMO) and orthogonal frequency division multiplexing (OFDM), we design a multi-stream AirComp scheme to fulfill the aggregation requirement of AirFL-LoRA. Furthermore, by deriving an optimality gap, we gain theoretical insights into the joint impact of rank selection and gradient aggregation distortion on the fine-tuning performance of AirFL-LoRA. Next, we formulate a non-convex problem to minimize the optimality gap, which is solved by the proposed backtracking-based alternating algorithm and the manifold optimization algorithm iteratively. Through fine-tuning LLMs for different downstream tasks, experimental results reveal that the AirFL-LoRA framework outperforms the state-of-the-art baselines on both training loss and perplexity, closely approximating the performance of FL with ideal aggregation.
Zitieren
APA
Sun, H., Tian, H., Ni, W., Zheng, J., Niyato, D., & Zhang, P. (2025). Federated Low-Rank Adaptation for Large Models Fine-Tuning Over Wireless Networks. IEEE Transactions on Wireless Communications, 24(1), 659-675. https://doi.org/10.1109/twc.2024.3497998
Vancouver
Sun H, Tian H, Ni W, Zheng J, Niyato D, Zhang P. Federated Low-Rank Adaptation for Large Models Fine-Tuning Over Wireless Networks. IEEE Transactions on Wireless Communications. 2025;24(1):659-675. doi:10.1109/twc.2024.3497998
BibTeX
@article{Sun2025,
author = {Sun, Haofeng and Tian, Hui and Ni, Wanli and Zheng, Jingheng and Niyato, Dusit and Zhang, Ping},
title = {Federated Low-Rank Adaptation for Large Models Fine-Tuning Over Wireless Networks},
journal = {IEEE Transactions on Wireless Communications},
year = {2025},
volume = {24},
number = {1},
pages = {659--675},
doi = {10.1109/twc.2024.3497998}
}
EUVIMED