EUVIMEDEUVIMED entdecken
Crossref

SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models

Han Yin, Yafeng Chen, Chong Deng, Luyao Cheng, Hui Wang, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (40) · 34467-34475

DOI: 10.1609/aaai.v40i40.40745 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Han Yin und 8 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Speaker Diarization and Recognition (SDR) task aims to predict ``who spoke when and what'' within an audio clip, which is a crucial task in various real-world multi-speaker scenarios such as meeting transcription and dialogue systems. Existing SDR systems typically adopt a cascaded framework, combining multiple modules such as speaker diarization (SD) and automatic speech recognition (ASR). The cascaded systems suffer from several limitations, such as error propagation, difficulty in handling overlapping speech, and lack of joint optimization for exploring the synergy between SD and ASR tasks. To address these limitations, we introduce SpeakerLM, a unified multimodal large language model for SDR that jointly performs SD and ASR in an end-to-end manner. Moreover, to facilitate diverse real-world scenarios, we incorporate a flexible speaker registration mechanism into SpeakerLM, enabling SDR under different speaker registration settings. SpeakerLM is progressively developed with a multi-stage training strategy on large-scale real data. Extensive experiments show that SpeakerLM demonstrates strong data scaling capability and generalizability, outperforming state-of-the-art cascaded baselines on both in-domain and out-of-domain public SDR benchmarks. Furthermore, experimental results show that the proposed speaker registration mechanism effectively ensures robust SDR performance of SpeakerLM across diverse speaker registration conditions and varying numbers of registered speakers.

Zitieren

APA

Yin, H., Chen, Y., Deng, C., Cheng, L., Wang, H., Tan, C., Chen, Q., Wang, W., & Li, X. (2026). SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 34467-34475. https://doi.org/10.1609/aaai.v40i40.40745

Vancouver

Yin H, Chen Y, Deng C, Cheng L, Wang H, Tan C, et al. SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(40):34467-34475. doi:10.1609/aaai.v40i40.40745

BibTeX
@article{Yin2026,
  author = {Yin, Han and Chen, Yafeng and Deng, Chong and Cheng, Luyao and Wang, Hui and Tan, Chao-Hong and Chen, Qian and Wang, Wen and Li, Xiangang},
  title = {SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {40},
  pages = {34467--34475},
  doi = {10.1609/aaai.v40i40.40745}
}
Kostenlos starten →