Recent Advances of Foundation Language Models-based Continual Learning: A Survey
ACM Computing Surveys · 31.05.2025 · 57 (5) · 1-38
DOI: 10.1145/3705725 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Computing Surveys, online erschienen am 09.01.2025. Verfasst von Yutao Yang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 48-mal zitiert (Crossref) und nennt 233 Literaturangaben. Gefördert von: National Science and Technology Major Project, National Nature Science Foundation of China, Science and Technology Commission of Shanghai Municipality.
Abstract (Originalsprache)
Recently, foundation language models (LMs) have marked significant achievements in the domains of natural language processing and computer vision. Unlike traditional neural network models, foundation LMs obtain a great ability for transfer learning by acquiring rich common sense knowledge through pre-training on extensive unsupervised datasets with a vast number of parameters. Despite these capabilities, LMs still struggle with catastrophic forgetting, hindering their ability to learn continuously like humans. To address this, continual learning (CL) methodologies have been introduced, allowing LMs to adapt to new tasks while retaining learned knowledge. However, a systematic taxonomy of existing approaches and a comparison of their performance are still lacking. In this article, we delve into a comprehensive review, summarization, and classification of the existing literature on CL-based approaches applied to foundation language models, such as pre-trained language models, large language models, and vision-language models. We divide these studies into offline and online CL, which consist of traditional methods, parameter-efficient-based methods, instruction tuning-based methods and continual pre-training methods. Additionally, we outline the typical datasets and metrics employed in CL research and provide a detailed analysis of the challenges and future work for LMs-based continual learning.
Förderung
- National Science and Technology Major Project
- National Nature Science Foundation of China
- Science and Technology Commission of Shanghai Municipality
- Shanghai Science and Technology Innovation Action Plan
Zitieren
APA
Yang, Y., Zhou, J., Ding, X., Huai, T., Liu, S., Chen, Q., Xie, Y., & He, L. (2025). Recent Advances of Foundation Language Models-based Continual Learning: A Survey. ACM Computing Surveys, 57(5), 1-38. https://doi.org/10.1145/3705725
Vancouver
Yang Y, Zhou J, Ding X, Huai T, Liu S, Chen Q, et al. Recent Advances of Foundation Language Models-based Continual Learning: A Survey. ACM Computing Surveys. 2025;57(5):1-38. doi:10.1145/3705725
BibTeX
@article{Yang2025,
author = {Yang, Yutao and Zhou, Jie and Ding, Xuanwen and Huai, Tianyu and Liu, Shunyu and Chen, Qin and Xie, Yuan and He, Liang},
title = {Recent Advances of Foundation Language Models-based Continual Learning: A Survey},
journal = {ACM Computing Surveys},
year = {2025},
volume = {57},
number = {5},
pages = {1--38},
doi = {10.1145/3705725}
}
EUVIMED