A lightweight approach to real-time speaker diarization: from audio toward audio-visual data streams
EURASIP Journal on Audio, Speech, and Music Processing · 28.11.2024 · 2024 (1)
DOI: 10.1186/s13636-024-00382-2 ↗Kurz gefasst
Fachartikel in der Zeitschrift EURASIP Journal on Audio, Speech, and Music Processing, online erschienen am 28.11.2024. Verfasst von Frantisek Kynych und 4 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 81 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Technology Agency of the Czech Republic, Technická Univerzita v Liberci, NTNU Norwegian University of Science and Technology.
Abstract (Originalsprache)
This manuscript deals with the task of real-time speaker diarization (SD) for stream-wise data processing. Therefore, in contrast to most of the existing papers, it considers not only the accuracy but also the computational demands of individual investigated methods. We first propose a new lightweight scheme allowing us to perform speaker diarization of streamed audio data. Our approach utilizes a modified residual network with squeeze-and-excitation blocks (SE-ResNet-34) to extract speaker embeddings in an optimized way using cached buffers. These embeddings are subsequently used for voice activity detection (VAD) and block-online k-means clustering with a look-ahead mechanism. The described scheme yields results similar to the reference offline system while operating solely on a CPU with a low real-time factor (RTF) below 0.1 and a constant latency of around 5.5 s. In the next part of the work, our research moves toward much more demanding and complex real-time processing of audio-visual data streams. For this purpose, we extend the above-mentioned scheme for audio data processing by adding an audio-video module. This module utilizes SyncNet combined with visual embeddings for identity tracking. Our resulting multi-modal SD framework then combines the outputs from audio and audio-video modules by using a new overlap-based fusion strategy. It yields diarization error rates that are competitive with the existing state-of-the-art offline audio-visual methods while allowing us to process various audio-video streams, e.g., from Internet or TV broadcasts, in real-time using GPU and with the same latency as for audio data processing.
Förderung
- Technology Agency of the Czech Republic
- Technická Univerzita v Liberci
- NTNU Norwegian University of Science and Technology
Zitieren
APA
Kynych, F., Cerva, P., Zdansky, J., Svendsen, T., & Salvi, G. (2024). A lightweight approach to real-time speaker diarization: from audio toward audio-visual data streams. EURASIP Journal on Audio, Speech, and Music Processing, 2024(1). https://doi.org/10.1186/s13636-024-00382-2
Vancouver
Kynych F, Cerva P, Zdansky J, Svendsen T, Salvi G. A lightweight approach to real-time speaker diarization: from audio toward audio-visual data streams. EURASIP Journal on Audio, Speech, and Music Processing. 2024;2024(1). doi:10.1186/s13636-024-00382-2
BibTeX
@article{Kynych2024,
author = {Kynych, Frantisek and Cerva, Petr and Zdansky, Jindrich and Svendsen, Torbjørn and Salvi, Giampiero},
title = {A lightweight approach to real-time speaker diarization: from audio toward audio-visual data streams},
journal = {EURASIP Journal on Audio, Speech, and Music Processing},
year = {2024},
volume = {2024},
number = {1},
doi = {10.1186/s13636-024-00382-2}
}
EUVIMED