Mitigating Catastrophic Forgetting in Fine-Tuned Large Language Models: An Experimental Study of LoRA and O-LoRA
Artificial Intelligence and Digital Technology · 2026
DOI: 10.70088/aa3p5364 ↗Kurz gefasst
Fachartikel in der Zeitschrift Artificial Intelligence and Digital Technology, 2026. Verfasst von Xinlan Zhang. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) have become a hot topic in AI, and since the GPT series they have achieved remarkable success across many domains. However, directly using a general-purpose model often fails to meet the needs of specific applications, which motivates fine-tuning with domain-specific data. Nevertheless, parameter-efficient fine-tuning (PEFT) methods such as LoRA may perform poorly on certain algorithmic benchmarks, raising concerns about cat-astrophic forgetting. In this paper, we conduct extensive experiments to confirm this phenomenon and investigate O-LoRA as a mitigation strategy. Results show that O-LoRA can effectively alleviate catastrophic forgetting under continual instruction fine-tuning, but its effectiveness can be sensitive to hyperparameters on some datasets. Overall, O-LoRA provides a practical direction for mitigating catastrophic forgetting during continual fine-tuning of LLMs.
Zitieren
APA
Zhang, X. (2026). Mitigating Catastrophic Forgetting in Fine-Tuned Large Language Models: An Experimental Study of LoRA and O-LoRA. Artificial Intelligence and Digital Technology. https://doi.org/10.70088/aa3p5364
Vancouver
Zhang X. Mitigating Catastrophic Forgetting in Fine-Tuned Large Language Models: An Experimental Study of LoRA and O-LoRA. Artificial Intelligence and Digital Technology. 2026. doi:10.70088/aa3p5364
BibTeX
@article{Zhang2026,
author = {Zhang, Xinlan},
title = {Mitigating Catastrophic Forgetting in Fine-Tuned Large Language Models: An Experimental Study of LoRA and O-LoRA},
journal = {Artificial Intelligence and Digital Technology},
year = {2026},
doi = {10.70088/aa3p5364}
}
EUVIMED