EUVIMEDEUVIMED entdecken
Crossref

L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific Regularization

Xiang Zhang, Rui Xie, Shikun Zhang

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (41) · 34817-34826

DOI: 10.1609/aaai.v40i41.40784 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xiang Zhang und 2 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Fine-tuning large language models (LLMs) in a parameter-efficient manner while preserving their pre-trained world knowledge remains a significant challenge. While Low-Rank Adaptation (LoRA) and its variants effectively mitigate catastrophic forgetting, they do not fully eliminate the loss of critical pre-trained knowledge. In this work, we first analyze the layer-wise distribution of domain-specific knowledge within LLMs through knowledge localization, and empirically identify a clear layer-specific pattern: pre-trained world knowledge predominantly resides in lower layers, whereas knowledge relevant to downstream tasks is more concentrated in higher layers. Motivated by this observation, we propose L2-LoRA, a simple yet effective variant of LoRA that applies layer-specific L2 regularization to the LoRA weights during fine-tuning. Specifically, L2-LoRA imposes stronger regularization on lower layers to preserve pre-trained world knowledge, while allowing greater adaptation in higher layers to better align with downstream tasks. Experiments across multiple benchmarks show that L2-LoRA not only consistently outperforms vanilla LoRA in downstream performance, but also effectively mitigates catastrophic forgetting by retaining more pre-trained knowledge.

Zitieren

APA

Zhang, X., Xie, R., & Zhang, S. (2026). L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific Regularization. Proceedings of the AAAI Conference on Artificial Intelligence, 40(41), 34817-34826. https://doi.org/10.1609/aaai.v40i41.40784

Vancouver

Zhang X, Xie R, Zhang S. L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific Regularization. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(41):34817-34826. doi:10.1609/aaai.v40i41.40784

BibTeX
@article{Zhang2026,
  author = {Zhang, Xiang and Xie, Rui and Zhang, Shikun},
  title = {L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific Regularization},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {41},
  pages = {34817--34826},
  doi = {10.1609/aaai.v40i41.40784}
}
Kostenlos starten →