HeLoRA: LoRA-heterogeneous Federated Fine-tuning for Foundation Models
ACM Transactions on Internet Technology · 31.05.2025 · 25 (2) · 1-22
DOI: 10.1145/3723877 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Internet Technology, online erschienen am 25.04.2025. Verfasst von Boyu Fan und 3 weiteren Autor:innen. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Guangzhou Municipal Nansha District Science and Technology Bureau, Nordic University Cooperation on Edge Intelligence.
Abstract (Originalsprache)
Foundation models (FMs) have achieved state-of-the-art performance across various domains, benefiting from their vast number of parameters and the extensive amount of publicly available training data. However, real-world deployments reveal challenges such as system heterogeneity, where not all devices can handle the complexity of FMs, and emerging privacy concerns that limit the availability of public data. To address these challenges, we propose HeLoRA, a novel approach combining low-rank adaptation (LoRA) with federated learning to enable heterogeneous federated fine-tuning. HeLoRA allows clients to fine-tune models with different complexities by adjusting the rank values of LoRA matrices, tailoring the process to each device’s capabilities. To tackle the challenge of aggregating models with different structures, HeLoRA introduces two variants, i.e., HeLoRA-Pad and HeLoRA-KD. HeLoRA-Pad employs context-based padding to standardize the LoRA matrices, aligning them with the global model through a rank-based adaptive aggregation strategy. In contrast, HeLoRA-KD leverages the idea of deep mutual learning for aggregation, allowing heterogeneous models to retain their original structures. Extensive experiments with various datasets and ablation studies demonstrate that HeLoRA outperforms existing baselines, promising to enhance the practical deployment of FMs in diverse real-world environments.
Förderung
- Guangzhou Municipal Nansha District Science and Technology Bureau
- Nordic University Cooperation on Edge Intelligence
Zitieren
APA
Fan, B., Su, X., Tarkoma, S., & Hui, P. (2025). HeLoRA: LoRA-heterogeneous Federated Fine-tuning for Foundation Models. ACM Transactions on Internet Technology, 25(2), 1-22. https://doi.org/10.1145/3723877
Vancouver
Fan B, Su X, Tarkoma S, Hui P. HeLoRA: LoRA-heterogeneous Federated Fine-tuning for Foundation Models. ACM Transactions on Internet Technology. 2025;25(2):1-22. doi:10.1145/3723877
BibTeX
@article{Fan2025,
author = {Fan, Boyu and Su, Xiang and Tarkoma, Sasu and Hui, Pan},
title = {HeLoRA: LoRA-heterogeneous Federated Fine-tuning for Foundation Models},
journal = {ACM Transactions on Internet Technology},
year = {2025},
volume = {25},
number = {2},
pages = {1--22},
doi = {10.1145/3723877}
}
EUVIMED