EUVIMEDEUVIMED entdecken
Crossref

Deep Learning-Based Multimodal Emotion Recognition Using Facial Expressions and Physiological Signals with Dynamic Fusion for Real-Time Human–Computer Interaction

G Shinde Akshata, S.G Shinde

International Journal of Recent Trends in Multidisciplinary Research · 27.07.2026 · 198

DOI: 10.59256/ijrtmr.20260604023 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Recent Trends in Multidisciplinary Research, online erschienen am 27.07.2026. Verfasst von G Shinde Akshata und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Emotion recognition has become an essential component of intelligent human–computer interaction, healthcare monitoring, driver assistance, and affective computing systems. Conventional facial expression-based emotion recognition methods often exhibit limited robustness because facial expressions may be intentionally suppressed, partially occluded, or inadequately represent an individual's actual emotional state. To address these limitations, this paper presents a deep learning-based multimodal emotion recognition framework that combines facial expression analysis with physiological signal processing to improve the reliability of real-time emotion classification. The proposed system employs a ResNet-18 convolutional neural network trained on the FER-2013 dataset for facial emotion recognition, while physiological information is acquired using a MAX30102 photoplethysmography (PPG) sensor and a Galvanic Skin Response (GSR) sensor interfaced with an STM32F411 microcontroller. The acquired physiological signals are processed to extract heart rate, heart rate variability (HRV), and skin conductance features that characterize the user's emotional arousal. A dynamic decision-level fusion strategy integrates facial prediction probabilities with physiological indicators using individualized baseline calibration, enabling effective recognition of concealed or ambiguous emotional states. The proposed framework operates in real time by combining embedded hardware, deep learning inference, and adaptive multimodal fusion within a unified architecture. Experimental investigations demonstrate that the multimodal approach provides more reliable emotion recognition than facial analysis alone, particularly for emotions associated with elevated physiological responses such as fear, stress, and surprise. Representative experimental results confirm stable real-time operation under different emotional conditions while maintaining computational efficiency suitable for practical deployment. Owing to its low-cost implementation, adaptive fusion mechanism, and real-time performance, the proposed framework is a promising solution for next-generation affective computing, intelligent healthcare, smart surveillance, wearable monitoring, and human–machine interaction applications.

Zitieren

APA

Akshata, G. S., & Shinde, S. (2026). Deep Learning-Based Multimodal Emotion Recognition Using Facial Expressions and Physiological Signals with Dynamic Fusion for Real-Time Human–Computer Interaction. International Journal of Recent Trends in Multidisciplinary Research, 198. https://doi.org/10.59256/ijrtmr.20260604023

Vancouver

Akshata GS, Shinde S. Deep Learning-Based Multimodal Emotion Recognition Using Facial Expressions and Physiological Signals with Dynamic Fusion for Real-Time Human–Computer Interaction. International Journal of Recent Trends in Multidisciplinary Research. 2026:198. doi:10.59256/ijrtmr.20260604023

BibTeX
@article{Akshata2026,
  author = {Akshata, G Shinde and Shinde, S.G},
  title = {Deep Learning-Based Multimodal Emotion Recognition Using Facial Expressions and Physiological Signals with Dynamic Fusion for Real-Time Human–Computer Interaction},
  journal = {International Journal of Recent Trends in Multidisciplinary Research},
  year = {2026},
  pages = {198},
  doi = {10.59256/ijrtmr.20260604023}
}
Kostenlos starten →