Random Dense Knowledge Distillation for Continual Learning
ACM Transactions on Multimedia Computing, Communications, and Applications · 30.06.2026 · 22 (6) · 1-21
DOI: 10.1145/3816731 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 23.06.2026. Verfasst von Jie Chu und 5 weiteren Autor:innen. Die Arbeit nennt 83 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Continual Learning (CL), involving sequential training on diverse tasks, often faces catastrophic forgetting. While knowledge distillation–based approaches exhibit notable success in preventing forgetting, we pinpoint a limitation in their ability to distill the cumulative knowledge of all the previous tasks. To remedy this, we propose Random Dense Knowledge Distillation (RDKD). RDKD uses a task pool to track the model’s capabilities. It partitions the output logits of the model into dense groups, each corresponding to a task in the task pool. It then distills all tasks’ knowledge using all groups. However, using all the groups can be computationally expensive, so we also suggest random group selection in each optimization step. Moreover, we propose an adaptive weighting scheme, which balances the learning of new classes and the retention of old classes, based on the count and similarity of the classes. Our RDKD outperforms recent state-of-the-art baselines across diverse benchmarks and scenarios. Empirical analysis underscores RDKD’s ability to enhance model stability, promotes flatter minima for improved generalization, and remains robust across various memory budgets and task orders. Moreover, it seamlessly integrates with other CL methods to boost performance and proves versatile in offline scenarios like model compression.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Chu, J., Liu, P., Su, T., Wu, Y., Song, Y., & Shi, Z. (2026). Random Dense Knowledge Distillation for Continual Learning. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(6), 1-21. https://doi.org/10.1145/3816731
Vancouver
Chu J, Liu P, Su T, Wu Y, Song Y, Shi Z. Random Dense Knowledge Distillation for Continual Learning. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(6):1-21. doi:10.1145/3816731
BibTeX
@article{Chu2026,
author = {Chu, Jie and Liu, Pei and Su, Tong and Wu, Yunpeng and Song, Yu and Shi, Zenglin},
title = {Random Dense Knowledge Distillation for Continual Learning},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2026},
volume = {22},
number = {6},
pages = {1--21},
doi = {10.1145/3816731}
}
EUVIMED