Order parameters and phase transitions of continual learning in deep neural networks
Proceedings of the National Academy of Sciences · 10.02.2026 · 123 (6)
DOI: 10.1073/pnas.2501899123 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, online erschienen am 06.02.2026. Verfasst von Haozhe Shan und 2 weiteren Autor:innen. Beteiligte Einrichtung: Center for Brain Science, Harvard University, Cambridge, MA 02138. Hauptthemen laut MEDLINE: Neural Networks, Computer, Learning. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 71 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: DOD | USN | Office of Naval Research (ONR), National Science Foundation (NSF).
Abstract (Originalsprache)
Continual learning (CL) enables animals to learn new tasks without erasing prior knowledge. CL in artificial neural networks (NNs) is challenging due to catastrophic forgetting, where new learning degrades performance on older tasks. While various techniques exist to mitigate forgetting, theoretical insights into when and why CL fails in NNs are lacking. Here, we present a statistical-mechanics theory of CL in deep, wide NNs, which characterizes the network’s input–output mapping as it learns a sequence of tasks. It gives rise to order parameters (OPs) that capture how task relations and network architecture influence forgetting and anterograde interference, as verified by numerical evaluations. For networks with a shared readout for all tasks (single-head CL), the relevant-feature and rule similarity between tasks, respectively measured by two OPs, are sufficient to predict a wide range of CL behaviors on classic benchmark tasks. In addition, the theory predicts that increasing the network depth can effectively reduce interference between tasks, thereby lowering forgetting. For networks with task-specific readouts (multihead CL), the theory identifies a phase transition where CL performance shifts dramatically as tasks become less similar, as measured by another task-similarity OP. While forgetting is relatively mild compared to single-head CL across all tasks, sufficiently low similarity leads to catastrophic anterograde interference, where the network retains old tasks and interpolates new training data perfectly but completely fails to generalize new learning. Our results delineate important factors affecting CL performance and offer theoretical insights into common heuristics for mitigation of forgetting.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Brain Science, Harvard University, Cambridge, MA 02138.
- Program in Neuroscience, Harvard Medical School, Boston, MA 02115.
- Department of Computer Science and Zuckerman Institute, Columbia University, New York, NY 10027.
- Biophysics Graduate Program, Harvard University, Cambridge, MA 02138.
- Edmond and Lily Safra Center for Brain Sciences, Hebrew University, Jerusalem 9190401, Israel.
ORCID-Profile: Haozhe Shan, Qianyi Li, Haim Sompolinsky
Förderung
- DOD | USN | Office of Naval Research (ONR)
- National Science Foundation (NSF)
Zitieren
APA
Shan, H., Li, Q., & Sompolinsky, H. (2026). Order parameters and phase transitions of continual learning in deep neural networks. Proceedings of the National Academy of Sciences, 123(6). https://doi.org/10.1073/pnas.2501899123
Vancouver
Shan H, Li Q, Sompolinsky H. Order parameters and phase transitions of continual learning in deep neural networks. Proceedings of the National Academy of Sciences. 2026;123(6). doi:10.1073/pnas.2501899123
BibTeX
@article{Shan2026,
author = {Shan, Haozhe and Li, Qianyi and Sompolinsky, Haim},
title = {Order parameters and phase transitions of continual learning in deep neural networks},
journal = {Proceedings of the National Academy of Sciences},
year = {2026},
volume = {123},
number = {6},
doi = {10.1073/pnas.2501899123}
}
EUVIMED