EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Zhenwei Zhang, Shengming Zhang iD, Dong Ni, Zhaoguo Wei, Kongjun Yang, Shan Jin, Gan Huang iD, Zhen Liang, Li Zhang, Linling Li, Huijun Ding iD, Zhiguo Zhang und 1 weitere

Sensors · 07.06.2024 · 24 (12) · 3714

DOI: 10.3390/s24123714 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 07.06.2024. Verfasst von Zhenwei Zhang und 12 weiteren Autor:innen. Beteiligte Einrichtung: School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China. Hauptthemen laut MEDLINE: Depression, Multimodal Imaging. Die Arbeit wurde bisher 75-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Shenzhen Science and Technology Research and Development Fund for Sustainable Development Project, Medical Scientific Research Foundation of Guangdong Province of China, Shenzhen Soft Science Research Program Project.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Depression is a major psychological disorder with a growing impact worldwide. Traditional methods for detecting the risk of depression, predominantly reliant on psychiatric evaluations and self-assessment questionnaires, are often criticized for their inefficiency and lack of objectivity. Advancements in deep learning have paved the way for innovations in depression risk detection methods that fuse multimodal data. This paper introduces a novel framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), designed to amalgamate auditory, visual, and textual cues for a comprehensive analysis of depression risk. Our approach encompasses three dedicated branches—Audio Branch, Video Branch, and Text Branch—each responsible for extracting salient features from the corresponding modality. These features are subsequently fused through a multimodal fusion (MMF) module, yielding a robust feature vector that feeds into a predictive modeling layer. To further our research, we devised an emotion elicitation paradigm based on two distinct tasks—reading and interviewing—implemented to gather a rich, sensor-based depression risk detection dataset. The sensory equipment, such as cameras, captures subtle facial expressions and vocal characteristics essential for our analysis. The research thoroughly investigates the data generated by varying emotional stimuli and evaluates the contribution of different tasks to emotion evocation. During the experiment, the AVTF-TBN model has the best performance when the data from the two tasks are simultaneously used for detection, where the F1 Score is 0.78, Precision is 0.76, and Recall is 0.81. Our experimental results confirm the validity of the paradigm and demonstrate the efficacy of the AVTF-TBN model in detecting depression risk, showcasing the crucial role of sensor-based data in mental health detection.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DepressionMultimodal ImagingRisk FactorsText MessagingVideo RecordingSound RecordingsHumansMaleFemaleYoung AdultAdultMiddle AgedDatasets as TopicEmotionsFacial Expression

Stichwörter der Autor:innen

depression risk detectionemotion elicitation paradigmmultimodal data

Autor:innen und Institutionen

  • School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China.
  • Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China.
  • Affiliated Mental Health Center, Southern University of Science and Technology, Shenzhen 518055, China.
  • Shenzhen Kangning Hospital, Shenzhen 518020, China.
  • Shenzhen Mental Health Center, Shenzhen 518020, China.
  • School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China.

ORCID-Profile: Shengming Zhang, Gan Huang, Huijun Ding

Förderung

  • Shenzhen Science and Technology Research and Development Fund for Sustainable Development Project
  • Medical Scientific Research Foundation of Guangdong Province of China
  • Shenzhen Soft Science Research Program Project

Zitieren

APA

Zhang, Z., Zhang, S., Ni, D., Wei, Z., Yang, K., Jin, S., Huang, G., Liang, Z., Zhang, L., Li, L., Ding, H., Zhang, Z., & Wang, J. (2024). Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data. Sensors, 24(12), 3714. https://doi.org/10.3390/s24123714

Vancouver

Zhang Z, Zhang S, Ni D, Wei Z, Yang K, Jin S, et al. Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data. Sensors. 2024;24(12):3714. doi:10.3390/s24123714

BibTeX
@article{Zhang2024,
  author = {Zhang, Zhenwei and Zhang, Shengming and Ni, Dong and Wei, Zhaoguo and Yang, Kongjun and Jin, Shan and Huang, Gan and Liang, Zhen and Zhang, Li and Li, Linling and Ding, Huijun and Zhang, Zhiguo and Wang, Jianhong},
  title = {Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data},
  journal = {Sensors},
  year = {2024},
  volume = {24},
  number = {12},
  pages = {3714},
  doi = {10.3390/s24123714}
}
Kostenlos starten →