EUVIMEDEUVIMED entdecken
Crossref

Deep Learning Algorithms for Multimodal Interaction Using Speech and Motion Data in Virtual Reality Systems

Ahmed Zubair, Fatima Al Rashed

PatternIQ Mining · 30.11.2024 · 1 (4)

DOI: 10.70023/sahd/241105 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PatternIQ Mining, online erschienen am 30.11.2024. Verfasst von Ahmed Zubair und 1 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal interaction (MMI) represented by speech and motion data (SMD) has enormous potential in virtual reality (VR) systems. However, real-time synchronization, context-sensitive interpretation, and effective fusion of heterogeneous data modalities remain open. The study presents a deep learning-based framework that fuses speech and motion data to provide better performance in interaction. This study proposes a novel method called MMI-CNNRNN that combines a Convolutional Neural Network (CNN) that features extraction in speech with a Recurrent Neural Network (RNN) for temporal motion analysis, integrated into a Transformer-based architecture to enhance the contextual understanding and responsiveness of the system. In this regard, the performance of the proposed framework is evaluated using benchmark multimodal datasets such as the IEMOCAP dataset. These results represent a 20% increase in interaction accuracy and a 15% latency reduction compared to unimodal and early fusion methods. The fusion of CNN and RNN mechanisms translates into more natural and intuitive interactions, making both the assistive device and the VR environment more adaptive and user-friendly. Concluding from the findings of the proposed work, efficient multimodal system development supports better accessibility and engagement among users with various needs.

Zitieren

APA

Zubair, A., & Rashed, F. A. (2024). Deep Learning Algorithms for Multimodal Interaction Using Speech and Motion Data in Virtual Reality Systems. PatternIQ Mining, 1(4). https://doi.org/10.70023/sahd/241105

Vancouver

Zubair A, Rashed FA. Deep Learning Algorithms for Multimodal Interaction Using Speech and Motion Data in Virtual Reality Systems. PatternIQ Mining. 2024;1(4). doi:10.70023/sahd/241105

BibTeX
@article{Zubair2024,
  author = {Zubair, Ahmed and Rashed, Fatima Al},
  title = {Deep Learning Algorithms for Multimodal Interaction Using Speech and Motion Data in Virtual Reality Systems},
  journal = {PatternIQ Mining},
  year = {2024},
  volume = {1},
  number = {4},
  doi = {10.70023/sahd/241105}
}
Kostenlos starten →