EUVIMEDEUVIMED entdecken
Crossref

TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model

Yixing Li, Ruobing Xie, Zhen Yang, Xingwu Sun, Shuaipeng Li, Weidong Han, Zhanhui Kang, Di Wang, Yu Cheng

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (38) · 31823-31833

DOI: 10.1609/aaai.v40i38.40451 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yixing Li und 8 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformers are the cornerstone of modern large language models, but their quadratic computational complexity limits efficiency in long-sequence processing. Recent advancements in Mamba, a state space model (SSM) with linear complexity, offer promising efficiency gains but suffer from unstable contextual learning and multitask generalization. Some works conduct layer-level hybrid structures that combine Transformer and Mamba layers, aiming to make full use of both advantages. This paper proposes TransMamba, a novel sequence-level hybrid framework that unifies Transformer and Mamba through shared parameter matrices (QKV and CBx), and thus could dynamically switch between attention and SSM mechanisms at different token lengths and layers. We design the Memory Converter to bridge Transformer and Mamba by converting attention outputs into SSM-compatible states, ensuring seamless information flow at TransPoints where the transformation happens. The TransPoint scheduling is also thoroughly explored for balancing effectiveness and efficiency. We conducted extensive experiments demonstrating that TransMamba achieves superior training efficiency and performance compared to single and hybrid baselines, and validated the deeper consistency between Transformer and Mamba paradigms at sequence level, offering a scalable solution for next-generation language modeling.

Zitieren

APA

Li, Y., Xie, R., Yang, Z., Sun, X., Li, S., Han, W., Kang, Z., Wang, D., & Cheng, Y. (2026). TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model. Proceedings of the AAAI Conference on Artificial Intelligence, 40(38), 31823-31833. https://doi.org/10.1609/aaai.v40i38.40451

Vancouver

Li Y, Xie R, Yang Z, Sun X, Li S, Han W, et al. TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(38):31823-31833. doi:10.1609/aaai.v40i38.40451

BibTeX
@article{Li2026,
  author = {Li, Yixing and Xie, Ruobing and Yang, Zhen and Sun, Xingwu and Li, Shuaipeng and Han, Weidong and Kang, Zhanhui and Wang, Di and Cheng, Yu},
  title = {TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {38},
  pages = {31823--31833},
  doi = {10.1609/aaai.v40i38.40451}
}
Kostenlos starten →