EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Efficient Differentially Private Fine-Tuning with QLoRA and Prefix Tuning for Large Language Models

Zhouyi Tan, Xi Xiong, Dong Xu

Journal of Computer Science and Artificial Intelligence · 27.03.2025 · 2 (3) · 50-54

DOI: 10.54097/we271q84 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Computer Science and Artificial Intelligence, online erschienen am 27.03.2025. Verfasst von Zhouyi Tan und 2 weiteren Autor:innen. Die Arbeit nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have achieved remarkable success in natural language processing (NLP) tasks. However, fine-tuning LLMs using private datasets raises significant privacy concerns, as models can inadvertently memorize sensitive information. Differentially Private Stochastic Gradient Descent (DP-SGD) provides a mathematically rigorous solution but suffers from high computational overhead, slow convergence, and excessive privacy budget consumption, making it impractical for large-scale models. To address these challenges, we propose an efficient differentially private fine-tuning method that combines Quantized Low-Rank Adaptation (QLoRA) and Prefix Tuning. QLoRA employs 4-bit NormalFloat quantization and low-rank adaptation, significantly reducing memory consumption and improving computational efficiency. Prefix Tuning optimizes a small set of prefix vectors without modifying the model’s main parameters, further reducing the impact of DP noise. Additionally, we introduce a hybrid adaptive gradient clipping strategy, which applies sample-wise adaptive clipping for Prefix Tuning and group-wise clipping for QLoRA, effectively balancing privacy protection and model utility. We evaluate our approach on GPT-2 using benchmark datasets including E2E NLG Challenge, XSum, SST-2, and DART, measuring performance using BLEU, ROUGE, and F1-score. Results demonstrate that QLoRA + Prefix Tuning achieves up to 75% memory reduction while maintaining over 95% of the original model performance under a moderate privacy budget (ε=3), outperforming traditional DP fine-tuning methods. Our work provides a practical and scalable solution for privacy-preserving LLM fine-tuning in resource-constrained environments.

Zitieren

APA

Tan, Z., Xiong, X., & Xu, D. (2025). Efficient Differentially Private Fine-Tuning with QLoRA and Prefix Tuning for Large Language Models. Journal of Computer Science and Artificial Intelligence, 2(3), 50-54. https://doi.org/10.54097/we271q84

Vancouver

Tan Z, Xiong X, Xu D. Efficient Differentially Private Fine-Tuning with QLoRA and Prefix Tuning for Large Language Models. Journal of Computer Science and Artificial Intelligence. 2025;2(3):50-54. doi:10.54097/we271q84

BibTeX
@article{Tan2025,
  author = {Tan, Zhouyi and Xiong, Xi and Xu, Dong},
  title = {Efficient Differentially Private Fine-Tuning with QLoRA and Prefix Tuning for Large Language Models},
  journal = {Journal of Computer Science and Artificial Intelligence},
  year = {2025},
  volume = {2},
  number = {3},
  pages = {50--54},
  doi = {10.54097/we271q84}
}
Kostenlos starten →