OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting During Parameter-Efficient Fine-Tuning
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (40) · 34088-34096
DOI: 10.1609/aaai.v40i40.40703 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yifeng Xiong und 1 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref).
Abstract (Originalsprache)
Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models but suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions that encode essential pre-trained knowledge. We propose Orthogonal Projection LoRA (OPLoRA), a theoretically grounded approach that prevents this interference through double-sided orthogonal projections. By decomposing frozen weights via SVD, OPLoRA constrains LoRA updates to lie entirely within the orthogonal complement of the top-k singular subspace using projections PL = I − Uk Ukᵀ and PR = I − Vk Vkᵀ. We prove that this construction exactly preserves the top-k singular triples, providing mathematical guarantees for knowledge retention. To quantify subspace interference, we introduce ρk, a metric measuring update alignment with dominant directions. Extensive experiments across commonsense reasoning, mathematics, and code generation demonstrate that OPLoRA significantly reduces forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B, establishing orthogonal projection as an effective mechanism for knowledge preservation in parameter-efficient fine-tuning.
Zitieren
APA
Xiong, Y., & Xie, X. (2026). OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting During Parameter-Efficient Fine-Tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 34088-34096. https://doi.org/10.1609/aaai.v40i40.40703
Vancouver
Xiong Y, Xie X. OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting During Parameter-Efficient Fine-Tuning. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(40):34088-34096. doi:10.1609/aaai.v40i40.40703
BibTeX
@article{Xiong2026,
author = {Xiong, Yifeng and Xie, Xiaohui},
title = {OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting During Parameter-Efficient Fine-Tuning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {40},
pages = {34088--34096},
doi = {10.1609/aaai.v40i40.40703}
}
EUVIMED