EUVIMEDEUVIMED entdecken
Crossref

Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models

Fei Wu, Jia Hu, Geyong Min, Shiqiang Wang

IEEE Transactions on Computers · 2026-04 · 75 (4) · 1650-1663

DOI: 10.1109/tc.2026.3655161 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Computers, 2026-04. Verfasst von Fei Wu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 45 Literaturangaben. Gefördert von: UK Engineering and Physical Sciences Research Council, Research and Innovation, HORIZON EUROPE Framework Programme.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Pre-trained Language Models (PLMs) have demonstrated their superiority and versatility in modern Natural Language Processing (NLP), effectively adapting to various downstream tasks through further fine-tuning. Federated Parameter-Efficient Fine-Tuning (FedPEFT) has emerged as a promising solution to address privacy and efficiency challenges in distributed training for PLMs on resource-constrained local devices. However, our measurements reveal two key limitations of FedPEFT: heterogeneous data across devices exacerbates performance degradation of low-rank adaptation, and a fixed parameter configuration results in communication inefficiency. To overcome these limitations, we propose FedARA, a novel adaptive rank allocation framework for federated parameter-efficient fine-tuning of language models. Specifically, FedARA employs truncated Singular Value Decomposition (SVD) adaptation to enhance similar feature representation across clients, significantly mitigating the adverse effects of data heterogeneity. Subsequently, it utilizes dynamic rank allocation to progressively identify critical ranks, effectively improving communication efficiency. Lastly, it leverages rank-based module pruning to automatically remove inactive modules, steadily reducing local computational cost and memory usage in each federated learning round. Extensive experiments show that FedARA consistently outperforms baselines by an average of 6.95% to 8.49% across various datasets and models under heterogeneous data while significantly improving communication efficiency by 2.40 $\boldsymbol{\times}$ × . Moreover, experiments on various edge devices demonstrate substantial decreases in total training time and energy consumption by up to 48.90% and 46.95%, respectively.

Förderung

  • UK Engineering and Physical Sciences Research Council
  • Research and Innovation
  • HORIZON EUROPE Framework Programme

Zitieren

APA

Wu, F., Hu, J., Min, G., & Wang, S. (2026). Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models. IEEE Transactions on Computers, 75(4), 1650-1663. https://doi.org/10.1109/tc.2026.3655161

Vancouver

Wu F, Hu J, Min G, Wang S. Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models. IEEE Transactions on Computers. 2026;75(4):1650-1663. doi:10.1109/tc.2026.3655161

BibTeX
@article{Wu2026,
  author = {Wu, Fei and Hu, Jia and Min, Geyong and Wang, Shiqiang},
  title = {Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models},
  journal = {IEEE Transactions on Computers},
  year = {2026},
  volume = {75},
  number = {4},
  pages = {1650--1663},
  doi = {10.1109/tc.2026.3655161}
}
Kostenlos starten →