Variational Data-Free Knowledge Distillation for Continual Learning
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-10 · 45 (10) · 12618-12634
DOI: 10.1109/tpami.2023.3271626 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-10. Verfasst von Xiaorong Li und 3 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 82 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
Deep neural networks suffer from catastrophic forgetting when trained on sequential tasks in continual learning. Various methods rely on storing data of previous tasks to mitigate catastrophic forgetting, which is prohibited in real-world applications considering privacy and security issues. In this paper, we consider a realistic setting of continual learning, where training data of previous tasks are unavailable and memory resources are limited. We contribute a novel knowledge distillation-based method in an information-theoretic framework by maximizing mutual information between outputs of previously learned and current networks. Due to the intractability of computation of mutual information, we instead maximize its variational lower bound, where the covariance of variational distribution is modeled by a graph convolutional network. The inaccessibility of data of previous tasks is tackled by Taylor expansion, yielding a novel regularizer in network training loss for continual learning. The regularizer relies on compressed gradients of network parameters. It avoids storing previous task data and previously learned networks. Additionally, we employ self-supervised learning technique for learning effective features, which improves the performance of continual learning. We conduct extensive experiments including image classification and semantic segmentation, and the results show that our method achieves state-of-the-art performance on continual learning benchmarks.
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Li, X., Wang, S., Sun, J., & Xu, Z. (2023). Variational Data-Free Knowledge Distillation for Continual Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), 12618-12634. https://doi.org/10.1109/tpami.2023.3271626
Vancouver
Li X, Wang S, Sun J, Xu Z. Variational Data-Free Knowledge Distillation for Continual Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(10):12618-12634. doi:10.1109/tpami.2023.3271626
BibTeX
@article{Li2023,
author = {Li, Xiaorong and Wang, Shipeng and Sun, Jian and Xu, Zongben},
title = {Variational Data-Free Knowledge Distillation for Continual Learning},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2023},
volume = {45},
number = {10},
pages = {12618--12634},
doi = {10.1109/tpami.2023.3271626}
}
EUVIMED