EUVIMEDEUVIMED entdecken
Crossref

DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling with Large Language Models

Yuanyuan Wang, Dongchao Yang, Yiwen Shao, Hangting Chen, Jiankun Zhao, Zhiyong Wu, Helen Meng, Xixin Wu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (40) · 33728-33736

DOI: 10.1609/aaai.v40i40.40663 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yuanyuan Wang und 7 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Extending pre-trained text Large Language Models (LLMs)’s speech understanding or generation abilities by introducing various effective speech tokens has attracted great attention in the speech research community. However, building a unified speech understanding and generation model still faces the following challenges: (1) Due to the huge modality gap between speech and text tokens, extending text LLMs to unified speech LLMs relies on large-scale paired data for fine-tuning, and (2) Generation and understanding tasks prefer information at different levels, e.g., generation benefits from detailed acoustic features, while understanding favors high-level semantics. This divergence leads to difficult performance optimization in one unified model. To solve these challenges, in this paper, we present two key insights in speech tokenization and speech language modeling. Specifically, we first propose an Understanding-driven Speech Tokenizer (USTokenizer), which extracts high-level semantic information essential for accomplishing understanding tasks using text LLMs. In this way, USToken enjoys better modality commonality with text, which reduces the difficulty of modality alignment in adapting text LLMs to speech LLMs. Secondly, we present DualSpeechLM, a dual-token modeling framework that concurrently models USToken as input and acoustic token as output within a unified, end-to-end framework, seamlessly integrating speech understanding and generation capabilities. Furthermore, we propose a novel semantic supervision loss and a Chain-of-Condition (CoC) strategy to stabilize model training and enhance speech generation performance. Experimental results demonstrate that our proposed approach effectively fosters a complementary relationship between understanding and generation tasks, highlighting the promising strategy of mutually enhancing both tasks in one unified model.

Zitieren

APA

Wang, Y., Yang, D., Shao, Y., Chen, H., Zhao, J., Wu, Z., Meng, H., & Wu, X. (2026). DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling with Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 33728-33736. https://doi.org/10.1609/aaai.v40i40.40663

Vancouver

Wang Y, Yang D, Shao Y, Chen H, Zhao J, Wu Z, et al. DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling with Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(40):33728-33736. doi:10.1609/aaai.v40i40.40663

BibTeX
@article{Wang2026,
  author = {Wang, Yuanyuan and Yang, Dongchao and Shao, Yiwen and Chen, Hangting and Zhao, Jiankun and Wu, Zhiyong and Meng, Helen and Wu, Xixin},
  title = {DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling with Large Language Models},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {40},
  pages = {33728--33736},
  doi = {10.1609/aaai.v40i40.40663}
}
Kostenlos starten →