EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy

Dengshi Li, Yu Gao iD, Chenyi Zhu, Qianrui Wang, Ruoxi Wang

Sensors · 2023

DOI: 10.3390/s23042053 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2023. Verfasst von Dengshi Li und 4 weiteren Autor:innen. Beteiligte Einrichtung: School of Artificial Intelligence, Jianghan University, Wuhan 430056, China. Hauptthemen laut MEDLINE: Lipreading, Speech Perception. Die Arbeit wurde bisher 39-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Science Foundation of China, Nature Science Foundation of Hubei Province, Doctoral Research Foundation of 376 Jianghan University.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The current accuracy of speech recognition can reach over 97% on different datasets, but in noisy environments, it is greatly reduced. Improving speech recognition performance in noisy environments is a challenging task. Due to the fact that visual information is not affected by noise, researchers often use lip information to help to improve speech recognition performance. This is where the performance of lip recognition and the effect of cross-modal fusion are particularly important. In this paper, we try to improve the accuracy of speech recognition in noisy environments by improving the lip reading performance and the cross-modal fusion effect. First, due to the same lip possibly containing multiple meanings, we constructed a one-to-many mapping relationship model between lips and speech allowing for the lip reading model to consider which articulations are represented from the input lip movements. Audio representations are also preserved by modeling the inter-relationships between paired audiovisual representations. At the inference stage, the preserved audio representations could be extracted from memory by the learned inter-relationships using only video input. Second, a joint cross-fusion model using the attention mechanism could effectively exploit complementary intermodal relationships, and the model calculates cross-attention weights on the basis of the correlations between joint feature representations and individual modalities. Lastly, our proposed model achieved a 4.0% reduction in WER in a −15 dB SNR environment compared to the baseline method, and a 10.1% reduction in WER compared to speech recognition. The experimental results show that our method could achieve a significant improvement over speech recognition models in different noise environments.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LipreadingSpeech PerceptionHumansSpeechLearningLip

Stichwörter der Autor:innen

audiovisual speech recognitioncross-modal fusionlip readingnoisy environment

Autor:innen und Institutionen

  • School of Artificial Intelligence, Jianghan University, Wuhan 430056, China.

ORCID-Profile: Yu Gao

Förderung

  • Natural Science Foundation of China
  • Nature Science Foundation of Hubei Province
  • Doctoral Research Foundation of 376 Jianghan University
  • Application Foundation Frontier Special Project of Wuhan Science 377 and Technology Plan Project
  • Application Foundation Frontier Special Project of Wuhan Science and Technology Plan Project
  • Doctoral Research Foundation of Jianghan University

Zitieren

APA

Li, D., Gao, Y., Zhu, C., Wang, Q., & Wang, R. (2023). Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy. Sensors. https://doi.org/10.3390/s23042053

Vancouver

Li D, Gao Y, Zhu C, Wang Q, Wang R. Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy. Sensors. 2023. doi:10.3390/s23042053

BibTeX
@article{Li2023,
  author = {Li, Dengshi and Gao, Yu and Zhu, Chenyi and Wang, Qianrui and Wang, Ruoxi},
  title = {Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23042053}
}
Kostenlos starten →