L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific Regularization
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (41) · 34817-34826
DOI: 10.1609/aaai.v40i41.40784 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xiang Zhang und 2 weiteren Autor:innen.
Abstract (Originalsprache)
Fine-tuning large language models (LLMs) in a parameter-efficient manner while preserving their pre-trained world knowledge remains a significant challenge. While Low-Rank Adaptation (LoRA) and its variants effectively mitigate catastrophic forgetting, they do not fully eliminate the loss of critical pre-trained knowledge. In this work, we first analyze the layer-wise distribution of domain-specific knowledge within LLMs through knowledge localization, and empirically identify a clear layer-specific pattern: pre-trained world knowledge predominantly resides in lower layers, whereas knowledge relevant to downstream tasks is more concentrated in higher layers. Motivated by this observation, we propose L2-LoRA, a simple yet effective variant of LoRA that applies layer-specific L2 regularization to the LoRA weights during fine-tuning. Specifically, L2-LoRA imposes stronger regularization on lower layers to preserve pre-trained world knowledge, while allowing greater adaptation in higher layers to better align with downstream tasks. Experiments across multiple benchmarks show that L2-LoRA not only consistently outperforms vanilla LoRA in downstream performance, but also effectively mitigates catastrophic forgetting by retaining more pre-trained knowledge.
Zitieren
APA
Zhang, X., Xie, R., & Zhang, S. (2026). L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific Regularization. Proceedings of the AAAI Conference on Artificial Intelligence, 40(41), 34817-34826. https://doi.org/10.1609/aaai.v40i41.40784
Vancouver
Zhang X, Xie R, Zhang S. L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific Regularization. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(41):34817-34826. doi:10.1609/aaai.v40i41.40784
BibTeX
@article{Zhang2026,
author = {Zhang, Xiang and Xie, Rui and Zhang, Shikun},
title = {L2-LoRA: Improving Low-Rank Adaptation with Layer-Specific Regularization},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {41},
pages = {34817--34826},
doi = {10.1609/aaai.v40i41.40784}
}
EUVIMED