EUVIMEDEUVIMED entdecken
Crossref

HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

Liyuan Wang, Jingyi Xie, Xingxing Zhang, Hang Su, Jun Zhu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-08 · 47 (8) · 6687-6702

DOI: 10.1109/tpami.2025.3562534 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-08. Verfasst von Liyuan Wang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 66 Literaturangaben. Gefördert von: NSFC Projects, Beijing Natural Science Foundation, Tsinghua Institute for Guo Qiang.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The deployment of pre-trained models (PTMs) has greatly advanced the field of continual learning (CL), enabling positive knowledge transfer and resilience to catastrophic forgetting. To sustain these advantages for sequentially arriving tasks, a promising direction involves keeping the pre-trained backbone frozen while employing parameter-efficient tuning (PET) techniques to instruct representation learning. Despite the popularity of Prompt-based PET for CL, its empirical design often leads to sub-optimal performance in our evaluation of different PTMs and target tasks. To this end, we propose a unified framework for CL with PTMs and PET that provides both theoretical and empirical advancements. We first perform an in-depth theoretical analysis of the CL objective in a pre-training context, decomposing it into hierarchical components namely within-task prediction, task-identity inference and task-adaptive prediction. We then present Hierarchical Decomposition PET (HiDe-PET), an innovative approach that explicitly optimizes the decomposed objective through incorporating task-specific and task-shared knowledge via mainstream PET techniques along with efficient recovery of pre-trained representations. Leveraging this framework, we delve into the distinct impacts of implementation strategy, PET technique and PET architecture, as well as adaptive knowledge accumulation amidst pronounced distribution changes. Finally, across various CL scenarios, our approach demonstrates remarkably superior performance over a broad spectrum of recent strong baselines.

Förderung

  • NSFC Projects
  • Beijing Natural Science Foundation
  • Tsinghua Institute for Guo Qiang
  • High Performance Computing Center, Tsinghua University
  • Postdoctoral Fellowship Program of CPSF
  • Shuimu Tsinghua Scholar
  • XPlorer Prize

Zitieren

APA

Wang, L., Xie, J., Zhang, X., Su, H., & Zhu, J. (2025). HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(8), 6687-6702. https://doi.org/10.1109/tpami.2025.3562534

Vancouver

Wang L, Xie J, Zhang X, Su H, Zhu J. HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(8):6687-6702. doi:10.1109/tpami.2025.3562534

BibTeX
@article{Wang2025,
  author = {Wang, Liyuan and Xie, Jingyi and Zhang, Xingxing and Su, Hang and Zhu, Jun},
  title = {HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {8},
  pages = {6687--6702},
  doi = {10.1109/tpami.2025.3562534}
}
Kostenlos starten →