Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning
ACM Transactions on Management Information Systems · 30.06.2025 · 16 (2) · 1-24
DOI: 10.1145/3682068 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Management Information Systems, online erschienen am 13.03.2025. Verfasst von Xiao-Yang Liu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 73-mal zitiert (Crossref) und nennt 65 Literaturangaben.
Abstract (Originalsprache)
The surge in interest and application of large language models (LLMs) has sparked a drive to fine-tune these models to suit specific applications, such as finance and medical science. However, concerns regarding data privacy have emerged, especially when multiple stakeholders aim to collaboratively enhance LLMs using sensitive data. In this scenario, federated learning becomes a natural choice, allowing decentralized fine-tuning without exposing raw data to central servers. Motivated by this, we investigate how data privacy can be ensured in LLM fine-tuning through practical federated learning approaches, enabling secure contributions from multiple parties to enhance LLMs. Yet, challenges arise: (1) despite avoiding raw data exposure, there is a risk of inferring sensitive information from model outputs, and (2) federated learning for LLMs incurs notable communication overhead. To address these challenges, this article introduces DP-LoRA, a novel federated learning algorithm tailored for LLMs. DP-LoRA preserves data privacy by employing a Gaussian mechanism that adds noise in weight updates, maintaining individual data privacy while facilitating collaborative model training. Moreover, DP-LoRA optimizes communication efficiency via low-rank adaptation, minimizing the transmission of updated weights during distributed training. The experimental results across medical, financial, and general datasets using various LLMs demonstrate that DP-LoRA effectively ensures strict privacy constraints while minimizing communication overhead.
Zitieren
APA
Liu, X., Zhu, R., Zha, D., Gao, J., Zhong, S., White, M., & Qiu, M. (2025). Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning. ACM Transactions on Management Information Systems, 16(2), 1-24. https://doi.org/10.1145/3682068
Vancouver
Liu X, Zhu R, Zha D, Gao J, Zhong S, White M, et al. Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning. ACM Transactions on Management Information Systems. 2025;16(2):1-24. doi:10.1145/3682068
BibTeX
@article{Liu2025,
author = {Liu, Xiao-Yang and Zhu, Rongyi and Zha, Daochen and Gao, Jiechao and Zhong, Shan and White, Matt and Qiu, Meikang},
title = {Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning},
journal = {ACM Transactions on Management Information Systems},
year = {2025},
volume = {16},
number = {2},
pages = {1--24},
doi = {10.1145/3682068}
}
EUVIMED