EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

ST‐LoRA: SVD‐Guided Sparse Low‐Rank Adaptation With Trainable Masks for Large Language and Vision Models

Quan Gan, Haifeng Hu, Feifei Wang, Ming Zhao, Xiaoyan Ren

IET Computer Vision · 2026-01 · 20 (1)

DOI: 10.1049/cvi2.70072 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IET Computer Vision, online erschienen am 25.06.2026. Verfasst von Quan Gan und 4 weiteren Autor:innen. Die Arbeit nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Humanities and Social Science Fund of Ministry of Education of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rapid advancement of large‐scale pre‐trained models in computer vision and natural language processing has necessitated the development of parameter‐efficient fine‐tuning (PEFT) techniques. Low‐Rank Adaptation (LoRA) has emerged as a leading approach, enabling efficient adaptation by decomposing weight updates into low‐rank matrices. However, standard LoRA suffers from inefficiencies due to uniform rank allocation, redundant parameter updates, and random initialisation. This paper introduces Sparse Truncated Low‐Rank Adaptation (ST‐LoRA), a novel PEFT method that enhances LoRA through three key innovations: (1) SVD‐based initialisation, leveraging singular value decomposition of pre‐trained weights to provide informed starting points for adaptation matrices. (2) Dynamic rank assignment, allocating layer‐specific ranks based on singular value distributions to optimise parameter efficiency. (3) Learnable sparse masks, selectively updating only the most critical connections to reduce computational overhead. We evaluate ST‐LoRA across vision (ViT‐L/16 on ImageNet‐1K, CIFAR‐100, and fine‐grained datasets) and language (LLaMA‐2 7B on GLUE and instruction‐following tasks) domains. Our comprehensive experiments include rigorous ablation studies, cross‐validation, and statistical significance testing. Our results demonstrate that ST‐LoRA consistently outperforms existing PEFT methods while significantly reducing computational costs. On ImageNet‐1K, ST‐LoRA achieves 84.0% top‐1 accuracy ( 0.12% std), surpassing LoRA ( r = 16) by 0.4% while using only 26.5% of its parameters and reducing training time by 57%. For language tasks, ST‐LoRA approaches full fine‐tuning performance with a fraction of the computational cost, reducing training time by up to 45% versus standard LoRA and 87% versus full fine‐tuning.

Förderung

  • Humanities and Social Science Fund of Ministry of Education of China

Zitieren

APA

Gan, Q., Hu, H., Wang, F., Zhao, M., & Ren, X. (2026). ST‐LoRA: SVD‐Guided Sparse Low‐Rank Adaptation With Trainable Masks for Large Language and Vision Models. IET Computer Vision, 20(1). https://doi.org/10.1049/cvi2.70072

Vancouver

Gan Q, Hu H, Wang F, Zhao M, Ren X. ST‐LoRA: SVD‐Guided Sparse Low‐Rank Adaptation With Trainable Masks for Large Language and Vision Models. IET Computer Vision. 2026;20(1). doi:10.1049/cvi2.70072

BibTeX
@article{Gan2026,
  author = {Gan, Quan and Hu, Haifeng and Wang, Feifei and Zhao, Ming and Ren, Xiaoyan},
  title = {ST‐LoRA: SVD‐Guided Sparse Low‐Rank Adaptation With Trainable Masks for Large Language and Vision Models},
  journal = {IET Computer Vision},
  year = {2026},
  volume = {20},
  number = {1},
  doi = {10.1049/cvi2.70072}
}
Kostenlos starten →