RRLoRA: Refactorized Low-Rank Adaptation With Learning-Rate Restarts for Efficient Fine-Tuning
IEEE Access · 2026 · 14 · 84922-84942
DOI: 10.1109/access.2026.3699357 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Mingzhe Yu und 1 weiteren Autor:innen. Die Arbeit nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: New Energy and Industrial Technology Development Organization, JST SPRING, Japan Society for the Promotion of Science.
Abstract (Originalsprache)
Low-rank adaptation (LoRA) has become a standard parameter-efficient fine-tuning technique for adapting large foundation models. However, prior work has shown that LoRA training dynamics and final performance are sensitive to initialization, and state-of-the-art data-driven initializations often incur significant runtime and engineering overhead. We propose RRLoRA, a simple training-time refactorization approach that improves LoRA fine-tuning without a separate initialization stage. RRLoRA starts from a standard random or weight-driven initialization, performs a short warmup fine-tuning phase, and then periodically refactorizes each adapter by computing a rank- $r$ truncated singular value decomposition of the adapter product and re-encoding the factors back into the LoRA matrices, coupled with a synchronized learning-rate hard restart to stabilize optimization. We further introduce an optional singular-value-based heuristic to calibrate the layer-wise LoRA scaling factor $\alpha $ under a fixed rank. Experiments across language understanding, reasoning, and open-ended vision question answering benchmarks show that RRLoRA improves performance in the vast majority of evaluated settings under matched training budgets. Representative gains include 0.38–1.03 percentage points in average accuracy on Commonsense170K, 0.03–3.93 percentage points in normalized exact-match accuracy on ChartQA, and 0.039–0.676 points in LLM-judge score on Kvasir-VQA-x1, while adding negligible wall-clock overhead and only minor changes in peak allocated GPU memory. The source code is available at https://github.com/UnbearableFate/rrlora_vqa
Förderung
- New Energy and Industrial Technology Development Organization
- JST SPRING
- Japan Society for the Promotion of Science
- JST CREST
- the Multidisciplinary Cooperative Research Program in CCS, University of Tsukuba
- Fujitsu Limited
Zitieren
APA
Yu, M., & Tatebe, O. (2026). RRLoRA: Refactorized Low-Rank Adaptation With Learning-Rate Restarts for Efficient Fine-Tuning. IEEE Access, 14, 84922-84942. https://doi.org/10.1109/access.2026.3699357
Vancouver
Yu M, Tatebe O. RRLoRA: Refactorized Low-Rank Adaptation With Learning-Rate Restarts for Efficient Fine-Tuning. IEEE Access. 2026;14:84922-84942. doi:10.1109/access.2026.3699357
BibTeX
@article{Yu2026,
author = {Yu, Mingzhe and Tatebe, Osamu},
title = {RRLoRA: Refactorized Low-Rank Adaptation With Learning-Rate Restarts for Efficient Fine-Tuning},
journal = {IEEE Access},
year = {2026},
volume = {14},
pages = {84922--84942},
doi = {10.1109/access.2026.3699357}
}
EUVIMED