EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Dmitry Ryumin iD, Denis Ivanko iD, Elena Ryumina iD

Sensors · 17.02.2023 · 23 (4) · 2284

DOI: 10.3390/s23042284 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 17.02.2023. Verfasst von Dmitry Ryumin und 2 weiteren Autor:innen. Beteiligte Einrichtung: St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, Russia. Hauptthemen laut MEDLINE: Gestures, Speech. Die Arbeit wurde bisher 96-mal zitiert (Crossref) und nennt 146 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Russian Science Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Audio-visual speech recognition (AVSR) is one of the most promising solutions for reliable speech recognition, particularly when audio is corrupted by noise. Additional visual information can be used for both automatic lip-reading and gesture recognition. Hand gestures are a form of non-verbal communication and can be used as a very important part of modern human–computer interaction systems. Currently, audio and video modalities are easily accessible by sensors of mobile devices. However, there is no out-of-the-box solution for automatic audio-visual speech and gesture recognition. This study introduces two deep neural network-based model architectures: one for AVSR and one for gesture recognition. The main novelty regarding audio-visual speech recognition lies in fine-tuning strategies for both visual and acoustic features and in the proposed end-to-end model, which considers three modality fusion approaches: prediction-level, feature-level, and model-level. The main novelty in gesture recognition lies in a unique set of spatio-temporal features, including those that consider lip articulation information. As there are no available datasets for the combined task, we evaluated our methods on two different large-scale corpora—LRW and AUTSL—and outperformed existing methods on both audio-visual speech recognition and gesture recognition tasks. We achieved AVSR accuracy for the LRW dataset equal to 98.76% and gesture recognition rate for the AUTSL dataset equal to 98.56%. The results obtained demonstrate not only the high performance of the proposed methodology, but also the fundamental possibility of recognizing audio-visual speech and gestures by sensors of mobile devices.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

GesturesSpeechHumansComputers, HandheldAcousticsComputer Systems

Stichwörter der Autor:innen

audio-visual speech recognitioncomputer visiondimensionality reduction techniquegesture recognitionlip-readingmodel-level fusionspatio-temporal features

Autor:innen und Institutionen

  • St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, Russia.

ORCID-Profile: Dmitry Ryumin, Denis Ivanko, Elena Ryumina

Förderung

  • Russian Science Foundation

Zitieren

APA

Ryumin, D., Ivanko, D., & Ryumina, E. (2023). Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices. Sensors, 23(4), 2284. https://doi.org/10.3390/s23042284

Vancouver

Ryumin D, Ivanko D, Ryumina E. Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices. Sensors. 2023;23(4):2284. doi:10.3390/s23042284

BibTeX
@article{Ryumin2023,
  author = {Ryumin, Dmitry and Ivanko, Denis and Ryumina, Elena},
  title = {Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices},
  journal = {Sensors},
  year = {2023},
  volume = {23},
  number = {4},
  pages = {2284},
  doi = {10.3390/s23042284}
}
Kostenlos starten →