EUVIMEDEUVIMED entdecken
Crossref

Federated Low-Rank Adaptation for Large Models Fine-Tuning Over Wireless Networks

Haofeng Sun, Hui Tian, Wanli Ni, Jingheng Zheng, Dusit Niyato, Ping Zhang

IEEE Transactions on Wireless Communications · 2025-01 · 24 (1) · 659-675

DOI: 10.1109/twc.2024.3497998 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Wireless Communications, 2025-01. Verfasst von Haofeng Sun und 5 weiteren Autor:innen. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 54 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The emergence of large language models (LLMs) with multi-task generalization capabilities is expected to improve the performance of artificial intelligence (AI)-as-a-service provision in 6G networks. By fine-tuning LLMs, AI services can become more precise and tailored to the demands of different downstream tasks. However, centralized fine-tuning paradigms pose a potential risk to user privacy, and existing distributed fine-tuning methods incur significant wireless transmission burdens due to the large-scale parameter transmission of LLMs. To tackle these challenges, by leveraging the low rank feature in LLM fine-tuning, we propose a wireless over-the-air federated learning (AirFL) based low-rank adaptation (LoRA) framework that integrates LoRA and over-the-air computation (AirComp) to achieve efficient fine-tuning and aggregation. Based on multiple-input multiple-output (MIMO) and orthogonal frequency division multiplexing (OFDM), we design a multi-stream AirComp scheme to fulfill the aggregation requirement of AirFL-LoRA. Furthermore, by deriving an optimality gap, we gain theoretical insights into the joint impact of rank selection and gradient aggregation distortion on the fine-tuning performance of AirFL-LoRA. Next, we formulate a non-convex problem to minimize the optimality gap, which is solved by the proposed backtracking-based alternating algorithm and the manifold optimization algorithm iteratively. Through fine-tuning LLMs for different downstream tasks, experimental results reveal that the AirFL-LoRA framework outperforms the state-of-the-art baselines on both training loss and perplexity, closely approximating the performance of FL with ideal aggregation.

Zitieren

APA

Sun, H., Tian, H., Ni, W., Zheng, J., Niyato, D., & Zhang, P. (2025). Federated Low-Rank Adaptation for Large Models Fine-Tuning Over Wireless Networks. IEEE Transactions on Wireless Communications, 24(1), 659-675. https://doi.org/10.1109/twc.2024.3497998

Vancouver

Sun H, Tian H, Ni W, Zheng J, Niyato D, Zhang P. Federated Low-Rank Adaptation for Large Models Fine-Tuning Over Wireless Networks. IEEE Transactions on Wireless Communications. 2025;24(1):659-675. doi:10.1109/twc.2024.3497998

BibTeX
@article{Sun2025,
  author = {Sun, Haofeng and Tian, Hui and Ni, Wanli and Zheng, Jingheng and Niyato, Dusit and Zhang, Ping},
  title = {Federated Low-Rank Adaptation for Large Models Fine-Tuning Over Wireless Networks},
  journal = {IEEE Transactions on Wireless Communications},
  year = {2025},
  volume = {24},
  number = {1},
  pages = {659--675},
  doi = {10.1109/twc.2024.3497998}
}
Kostenlos starten →