Grow-on-Demand: Sparse and Adaptive Expert Expansion for Continual Instruction Tuning
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (34) · 28474-28482
DOI: 10.1609/aaai.v40i34.40077 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Ying Zhang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Continual instruction tuning aims to incrementally adapt large language models to new tasks without forgetting previously acquired knowledge. Existing approaches often struggle to balance plasticity and stability. Replay-based methods retrain on historical data, which raises privacy concerns. Architecture-based methods allocate task-specific components, resulting in significant parameter growth. To address this, we consider a structure-sharing strategy that enables parameter reuse across similar tasks and expands only when necessary, avoiding any data replay. Specifically, we introduce Grow-on-Demand (GoD-MoE), a parameter-efficient framework that is based on sparse and adaptive expert module expansion for continual instruction tuning. GoD-MoE inserts multiple LoRA-based experts into attention layers and dynamically activates a small subset of experts for each task. To avoid redundant parameter growth, we develop an Expert Demand Detector that determines whether new experts are added, facilitating adaptive structural sharing and minimizing parameter overhead. We conduct comprehensive experiments on the TRACE benchmark, demonstrating that GoD-MoE achieves state-of-the-art performance. Furthermore, it effectively mitigates catastrophic forgetting and even outperforms several advanced replay-based baselines.
Zitieren
APA
Zhang, Y., Guo, X., Zhao, Y., Sui, X., Zhou, B., Qian, X., & Yuan, X. (2026). Grow-on-Demand: Sparse and Adaptive Expert Expansion for Continual Instruction Tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(34), 28474-28482. https://doi.org/10.1609/aaai.v40i34.40077
Vancouver
Zhang Y, Guo X, Zhao Y, Sui X, Zhou B, Qian X, et al. Grow-on-Demand: Sparse and Adaptive Expert Expansion for Continual Instruction Tuning. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(34):28474-28482. doi:10.1609/aaai.v40i34.40077
BibTeX
@article{Zhang2026,
author = {Zhang, Ying and Guo, Xingyue and Zhao, Yu and Sui, Xuhui and Zhou, Baohang and Qian, Xinying and Yuan, Xiaojie},
title = {Grow-on-Demand: Sparse and Adaptive Expert Expansion for Continual Instruction Tuning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {34},
pages = {28474--28482},
doi = {10.1609/aaai.v40i34.40077}
}
EUVIMED