Multimodal depression recognition and analysis: Facial expression and body posture changes via emotional stimuli
Journal of Affective Disorders · 2025
DOI: 10.1016/j.jad.2025.03.155 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Affective Disorders, 2025. Verfasst von Yang Liu und 8 weiteren Autor:innen. Beteiligte Einrichtung: Endocrinology, The Fifth Clinical College of Guangzhou University of Chinese Medicine, Guangzhou 510095, China; Endocrinology, Guangdong Provincial S…. Hauptthemen laut MEDLINE: Facial Expression, Posture, Emotions, Deep Learning. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 52 Literaturangaben. Gefördert von: Ministry of Education of the People's Republic of China, Key Technology Research and Development Program of Shandong, Natural Science Foundation of Shandong Province.
Themen: Depression
Abstract (Originalsprache)
BACKGROUND: Clinical studies have shown that facial expressions and body posture in depressed patients differ significantly from those of healthy individuals. Combining relevant behavioral features with artificial intelligence technology can effectively improve the efficiency of depression detection, thereby assisting doctors in early identification of patients. This study aims to develop an end-to-end multimodal recognition model combining facial expressions and body posture via deep learning techniques, enabling rapid preliminary screening of depression. METHODS: We invited 146 subjects (73 in the patient group and 73 in the control group) to participate in an emotion-stimulus experiment for depression recognition. We focused on differentiating depression patients from the control group by analyzing changes in body posture and facial expressions under emotional stimuli. We first extracted images of body position and facial emotions from the video, then used a pre-trained ResNet-50 network to extract features. Additionally, we analyzed facial expression features using OpenFace for sequence analysis. Subsequently, various deep learning frameworks were combined to assess the severity of depression. RESULTS: We found that under different stimuli, facial expression units AU04, AU07, AU10, AU12, AU17, and AU26 had significant effects in the emotion-stimulus experiment, with these features generally being negative. The decision-level fusion model based on facial expressions and body posture achieved excellent results, with the highest accuracy of 0.904 and an F1 score of 0.901. CONCLUSIONS: The experimental results suggest that depression patients exhibit predominantly negative facial expressions. This study validates the emotion-stimulus experiment, demonstrating that combining facial expressions and body posture enables accurate preliminary depression screening.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Endocrinology, The Fifth Clinical College of Guangzhou University of Chinese Medicine, Guangzhou 510095, China; Endocrinology, Guangdong Provincial Second Hospital of Traditional Chinese Medicine, Guangzhou 510095, Chin…
- Gansu Provincial Key Laboratory of Wearable Computing, School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China.
- Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Qilu University of Technology (Shandong Academy of Science), Jinan 250353, China.
- Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Qilu University of Technology (Shandong Academy of Science), Jinan 250353, China. Electronic address: wangqx@qlu.edu.cn.
- College of Traditional Chinese Medicine, Shandong University of Traditional Chinese Medicine, Jinan 250355, China. Electronic address: yhysdutcm@163.com.
- Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Qilu University of Technology (Shandong Academy of Science), Jinan 250353, China; School of Computer Science, Guangdong Universi…
Förderung
- Ministry of Education of the People's Republic of China
- Key Technology Research and Development Program of Shandong
- Natural Science Foundation of Shandong Province
Zitieren
APA
Liu, Y., Li, X., Wang, M., Bi, J., Lin, S., Wang, Q., Yu, Y., Ye, J., & Zheng, Y. (2025). Multimodal depression recognition and analysis: Facial expression and body posture changes via emotional stimuli. Journal of Affective Disorders. https://doi.org/10.1016/j.jad.2025.03.155
Vancouver
Liu Y, Li X, Wang M, Bi J, Lin S, Wang Q, et al. Multimodal depression recognition and analysis: Facial expression and body posture changes via emotional stimuli. Journal of Affective Disorders. 2025. doi:10.1016/j.jad.2025.03.155
BibTeX
@article{Liu2025,
author = {Liu, Yang and Li, Xingyun and Wang, Mengqi and Bi, Jianlu and Lin, Shaoqin and Wang, Qingxiang and Yu, Yanhong and Ye, Jiayu and Zheng, Yunshao},
title = {Multimodal depression recognition and analysis: Facial expression and body posture changes via emotional stimuli},
journal = {Journal of Affective Disorders},
year = {2025},
doi = {10.1016/j.jad.2025.03.155}
}
EUVIMED