EUVIMEDEUVIMED entdecken
Crossref

Scale Regularization for Stable Low-Rank Adaptation

Tan Xeng Ian

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41528-41530

DOI: 10.1609/aaai.v40i48.42329 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Tan Xeng Ian.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Low-Rank Adaptation (LoRA) has emerged as a practical and efficient method for fine-tuning large language models under limited computational budgets. However, recent studies have shown that LoRA can suffer from training instability when applied to models with large embedding dimensions, due to the imbalanced in magnitudes between its low-rank matrices. In this work, we propose a novel regularization strategy that stabilizes LoRA training by penalizing logarithmic magnitude differences between the low-rank matrices, showing theoretically that it should lead to efficient feature learning. We further propose evaluation methods to systematically assess training stability and performance of our proposed solution along with other LoRA variants.

Zitieren

APA

Ian, T. X. (2026). Scale Regularization for Stable Low-Rank Adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41528-41530. https://doi.org/10.1609/aaai.v40i48.42329

Vancouver

Ian TX. Scale Regularization for Stable Low-Rank Adaptation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41528-41530. doi:10.1609/aaai.v40i48.42329

BibTeX
@article{Ian2026,
  author = {Ian, Tan Xeng},
  title = {Scale Regularization for Stable Low-Rank Adaptation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {48},
  pages = {41528--41530},
  doi = {10.1609/aaai.v40i48.42329}
}
Kostenlos starten →