EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator

Luoming Zhang, Zhenyu Lou, Yangwei Ying, Cheng Yang, Hong Zhou

Applied Sciences · 26.12.2024 · 15 (1) · 82

DOI: 10.3390/app15010082 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 26.12.2024. Verfasst von Luoming Zhang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research and Development Program of China, Experimental Technology Reaseach Project of Zhejiang University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we present a Low-Rank Gradient Estimator (LoGE) to accelerate the finetune-time computation of transformers, especially large language models (LLMs). Unlike Parameter-Efficient Fine-Tuning (PEFT) methods, which primarily aim to minimize the number of fine-tuning parameters, LoGE also significantly reduces the computational load of activation gradient calculations by decomposing pre-trained weights and utilizing low-rank matrices during the backward pass. Our approach includes an effective solution for identifying sensitive and important latent subspaces in large models before training with downstream datasets. As LoGE does not alter the network structure, it can be conveniently integrated into existing models. We validated LoGE’s efficacy through comprehensive experiments across various models on various tasks. For the widely used LLaMA model equipped with LoRA, LoGE achieves up to a 1.3× speedup while maintaining graceful accuracy.

Förderung

  • National Key Research and Development Program of China
  • Experimental Technology Reaseach Project of Zhejiang University

Zitieren

APA

Zhang, L., Lou, Z., Ying, Y., Yang, C., & Zhou, H. (2024). Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator. Applied Sciences, 15(1), 82. https://doi.org/10.3390/app15010082

Vancouver

Zhang L, Lou Z, Ying Y, Yang C, Zhou H. Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator. Applied Sciences. 2024;15(1):82. doi:10.3390/app15010082

BibTeX
@article{Zhang2024,
  author = {Zhang, Luoming and Lou, Zhenyu and Ying, Yangwei and Yang, Cheng and Zhou, Hong},
  title = {Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator},
  journal = {Applied Sciences},
  year = {2024},
  volume = {15},
  number = {1},
  pages = {82},
  doi = {10.3390/app15010082}
}
Kostenlos starten →