Real-time multilingual speech recognition and speaker diarization system based on Whisper segmentation
PeerJ Computer Science · 29.03.2024 · 10 · e1973
DOI: 10.7717/peerj-cs.1973 ↗Kurz gefasst
Fachartikel in der Zeitschrift PeerJ Computer Science, online erschienen am 29.03.2024. Verfasst von Ke-Ming Lyu und 2 weiteren Autor:innen. Beteiligte Einrichtung: Computer Science and Information Engineering, Chang Gung University, Taoyuan, Taiwan. Die Arbeit wurde bisher 30-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science and Technology Council, Chang Gung Memorial Hospital.
Abstract (Originalsprache)
This research presents the development of a cutting-edge real-time multilingual speech recognition and speaker diarization system that leverages OpenAI’s Whisper model. The system specifically addresses the challenges of automatic speech recognition (ASR) and speaker diarization (SD) in dynamic, multispeaker environments, with a focus on accurately processing Mandarin speech with Taiwanese accents and managing frequent speaker switches. Traditional speech recognition systems often fall short in such complex multilingual and multispeaker contexts, particularly in SD. This study, therefore, integrates advanced speech recognition with speaker diarization techniques optimized for real-time applications. These optimizations include handling model outputs efficiently and incorporating speaker embedding technology. The system was evaluated using data from Taiwanese talk shows and political commentary programs, featuring 46 diverse speakers. The results showed a promising word diarization error rate (WDER) of 2.68% in two-speaker scenarios and 11.65% in three-speaker scenarios, with an overall WDER of 6.96%. This performance is comparable to that of non-real-time baseline models, highlighting the system’s ability to adapt to various complex conversational dynamics, a significant advancement in the field of real-time multilingual speech processing.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Computer Science and Information Engineering, Chang Gung University, Taoyuan, Taiwan.
- Physical Medicine and Rehabilitation, Chang Gung Memorial Hospital, Taoyuan, Taiwan.
- Bachelor Program in Artificial Intelligence, Chang Gung University, Taoyuan, Taiwan.
ORCID-Profile: Hsien-Tsung Chang
Förderung
- National Science and Technology Council
- Chang Gung Memorial Hospital
Zitieren
APA
Lyu, K. M., Lyu, R. Y., & Chang, H. T. (2024). Real-time multilingual speech recognition and speaker diarization system based on Whisper segmentation. PeerJ Computer Science, 10, e1973. https://doi.org/10.7717/peerj-cs.1973
Vancouver
Lyu KM, Lyu RY, Chang HT. Real-time multilingual speech recognition and speaker diarization system based on Whisper segmentation. PeerJ Computer Science. 2024;10:e1973. doi:10.7717/peerj-cs.1973
BibTeX
@article{Lyu2024,
author = {Lyu, Ke-Ming and Lyu, Ren-Yuan and Chang, Hsien-Tsung},
title = {Real-time multilingual speech recognition and speaker diarization system based on Whisper segmentation},
journal = {PeerJ Computer Science},
year = {2024},
volume = {10},
pages = {e1973},
doi = {10.7717/peerj-cs.1973}
}
EUVIMED