A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model
Sensors · 2023
DOI: 10.3390/s23218770 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, 2023. Verfasst von Guizhu Li und 4 weiteren Autor:innen. Beteiligte Einrichtung: College of Electronic Engineering, Ocean University of China, Qingdao 266100, China. Hauptthemen laut MEDLINE: Speech, Speech Perception. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: the Key Research and Development Projects in Hainan Province, the National Natural Science Foundation of China, the Hainan Province of China.
Abstract (Originalsprache)
The cocktail party problem can be more effectively addressed by leveraging the speaker’s visual and audio information. This paper proposes a method to improve the audio’s separation using two visual cues: facial features and lip movement. Firstly, residual connections are introduced in the audio separation module to extract detailed features. Secondly, considering the video stream contains information other than the face, which has a minimal correlation with the audio, an attention mechanism is employed in the face module to focus on crucial information. Then, the loss function considers the audio-visual similarity to take advantage of the relationship between audio and visual completely. Experimental results on the public VoxCeleb2 dataset show that the proposed model significantly enhanced SDR, PSEQ, and STOI, especially 4 dB improvements in SDR.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- College of Electronic Engineering, Ocean University of China, Qingdao 266100, China.
- Sanya Oceanography Institution, Ocean University of China, Sanya 572024, China.
- College of Automation and Electronic Engineering, Qingdao University of Science and Technology, Qingdao 266061, China.
Förderung
- the Key Research and Development Projects in Hainan Province
- the National Natural Science Foundation of China
- the Hainan Province of China
- the Shandong Provincial Natural Science Foundation
- Key Research and Development Projects in Hainan Province
- Shandong Provincial Natural Science Foundation
- National Natural Science Foundation of China
- Hainan Province of China
Zitieren
APA
Li, G., Fu, M., Sun, M., Liu, X., & Zheng, B. (2023). A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model. Sensors. https://doi.org/10.3390/s23218770
Vancouver
Li G, Fu M, Sun M, Liu X, Zheng B. A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model. Sensors. 2023. doi:10.3390/s23218770
BibTeX
@article{Li2023,
author = {Li, Guizhu and Fu, Min and Sun, Mengnan and Liu, Xuefeng and Zheng, Bing},
title = {A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model},
journal = {Sensors},
year = {2023},
doi = {10.3390/s23218770}
}
EUVIMED