EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model

Guizhu Li, Min Fu, Mengnan Sun, Xuefeng Liu, Bing Zheng

Sensors · 2023

DOI: 10.3390/s23218770 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2023. Verfasst von Guizhu Li und 4 weiteren Autor:innen. Beteiligte Einrichtung: College of Electronic Engineering, Ocean University of China, Qingdao 266100, China. Hauptthemen laut MEDLINE: Speech, Speech Perception. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: the Key Research and Development Projects in Hainan Province, the National Natural Science Foundation of China, the Hainan Province of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The cocktail party problem can be more effectively addressed by leveraging the speaker’s visual and audio information. This paper proposes a method to improve the audio’s separation using two visual cues: facial features and lip movement. Firstly, residual connections are introduced in the audio separation module to extract detailed features. Secondly, considering the video stream contains information other than the face, which has a minimal correlation with the audio, an attention mechanism is employed in the face module to focus on crucial information. Then, the loss function considers the audio-visual similarity to take advantage of the relationship between audio and visual completely. Experimental results on the public VoxCeleb2 dataset show that the proposed model significantly enhanced SDR, PSEQ, and STOI, especially 4 dB improvements in SDR.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

SpeechSpeech PerceptionLipMovementCues

Stichwörter der Autor:innen

U-Netattention mechanismaudio-visualspeech separation

Autor:innen und Institutionen

  • College of Electronic Engineering, Ocean University of China, Qingdao 266100, China.
  • Sanya Oceanography Institution, Ocean University of China, Sanya 572024, China.
  • College of Automation and Electronic Engineering, Qingdao University of Science and Technology, Qingdao 266061, China.

Förderung

  • the Key Research and Development Projects in Hainan Province
  • the National Natural Science Foundation of China
  • the Hainan Province of China
  • the Shandong Provincial Natural Science Foundation
  • Key Research and Development Projects in Hainan Province
  • Shandong Provincial Natural Science Foundation
  • National Natural Science Foundation of China
  • Hainan Province of China

Zitieren

APA

Li, G., Fu, M., Sun, M., Liu, X., & Zheng, B. (2023). A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model. Sensors. https://doi.org/10.3390/s23218770

Vancouver

Li G, Fu M, Sun M, Liu X, Zheng B. A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model. Sensors. 2023. doi:10.3390/s23218770

BibTeX
@article{Li2023,
  author = {Li, Guizhu and Fu, Min and Sun, Mengnan and Liu, Xuefeng and Zheng, Bing},
  title = {A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23218770}
}
Kostenlos starten →