EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 3.0

Development of a Multimodal Mental Health Monitoring System Using Emotion Recognition and HRV with AI Chatbot

Younghyun Ko, Jong Hyeok Han, Gook Hwan Yeom, Hee-Jae Jeon, Hyeon-Ki Jang, Byeong Hee Kim, Yong-Jai Park

Journal of the Korean Society for Precision Engineering · 01.06.2026 · 43 (6) · 567-577

DOI: 10.7736/jkspe.025.00023 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the Korean Society for Precision Engineering, erschienen am 01.06.2026. Verfasst von Younghyun Ko und 6 weiteren Autor:innen. Die Arbeit nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 3.0). Gefördert von: Ministry of Trade, Industry and Resources, Korea Institute for Advancement of Technology.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Despite the increasing focus on the mental health of older adults and active senior populations, assessment tools still lag behind those for physical health monitoring. To bridge this gap, this study introduces an AI chatbot-based multimodal stress monitoring system that utilizes emotion recognition and heart rate variability (HRV). The system analyzes chatbot conversations, video, audio, and heart rate signals to assess facial expressions, speech emotions, and HRV, allowing for stress evaluation and user stratification into risk groups. Negative emotions are quantified and combined with HRV data to generate a stress score. Facial and speech emotion models were trained on the RAVDESS, CREMA, and TESS datasets, yielding 21,000 augmented samples through a BiLSTM network. Additionally, a deep learning-based HRV model utilized data from smartwatches to predict stress levels. By integrating facial, vocal, and HRV features through weighted fusion, the system produces a comprehensive stress index that categorizes users Healthy, Caution, Risk. This approach facilitates continuous monitoring at home, supporting early detection for preventive care and informed clinical decision-making.

Förderung

  • Ministry of Trade, Industry and Resources
  • Korea Institute for Advancement of Technology

Zitieren

APA

Ko, Y., Han, J. H., Yeom, G. H., Jeon, H., Jang, H., Kim, B. H., & Park, Y. (2026). Development of a Multimodal Mental Health Monitoring System Using Emotion Recognition and HRV with AI Chatbot. Journal of the Korean Society for Precision Engineering, 43(6), 567-577. https://doi.org/10.7736/jkspe.025.00023

Vancouver

Ko Y, Han JH, Yeom GH, Jeon H, Jang H, Kim BH, et al. Development of a Multimodal Mental Health Monitoring System Using Emotion Recognition and HRV with AI Chatbot. Journal of the Korean Society for Precision Engineering. 2026;43(6):567-577. doi:10.7736/jkspe.025.00023

BibTeX
@article{Ko2026,
  author = {Ko, Younghyun and Han, Jong Hyeok and Yeom, Gook Hwan and Jeon, Hee-Jae and Jang, Hyeon-Ki and Kim, Byeong Hee and Park, Yong-Jai},
  title = {Development of a Multimodal Mental Health Monitoring System Using Emotion Recognition and HRV with AI Chatbot},
  journal = {Journal of the Korean Society for Precision Engineering},
  year = {2026},
  volume = {43},
  number = {6},
  pages = {567--577},
  doi = {10.7736/jkspe.025.00023}
}
Kostenlos starten →