Scale Regularization for Stable Low-Rank Adaptation
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41528-41530
DOI: 10.1609/aaai.v40i48.42329 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Tan Xeng Ian.
Abstract (Originalsprache)
Low-Rank Adaptation (LoRA) has emerged as a practical and efficient method for fine-tuning large language models under limited computational budgets. However, recent studies have shown that LoRA can suffer from training instability when applied to models with large embedding dimensions, due to the imbalanced in magnitudes between its low-rank matrices. In this work, we propose a novel regularization strategy that stabilizes LoRA training by penalizing logarithmic magnitude differences between the low-rank matrices, showing theoretically that it should lead to efficient feature learning. We further propose evaluation methods to systematically assess training stability and performance of our proposed solution along with other LoRA variants.
Zitieren
APA
Ian, T. X. (2026). Scale Regularization for Stable Low-Rank Adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41528-41530. https://doi.org/10.1609/aaai.v40i48.42329
Vancouver
Ian TX. Scale Regularization for Stable Low-Rank Adaptation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41528-41530. doi:10.1609/aaai.v40i48.42329
BibTeX
@article{Ian2026,
author = {Ian, Tan Xeng},
title = {Scale Regularization for Stable Low-Rank Adaptation},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {48},
pages = {41528--41530},
doi = {10.1609/aaai.v40i48.42329}
}
EUVIMED