EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Predicting depression by using a novel deep learning model and video-audio-text multimodal data

Yifu Li, Xueping Yang, Meng Zhao, Jiangtao Wang, Yudong Yao, Wei Qian, Shouliang Qi

Frontiers in Psychiatry · 24.09.2025 · 16

DOI: 10.3389/fpsyt.2025.1602650 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Psychiatry, online erschienen am 24.09.2025. Verfasst von Yifu Li und 6 weiteren Autor:innen. Beteiligte Einrichtung: College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Depression is a prevalent mental health disorder affecting millions of people. Traditional diagnostic methods primarily rely on self-reported questionnaires and clinical interviews, which can be subjective and vary significantly between individuals. This paper introduces the Integrative Multimodal Depression Detection Network (IMDD-Net), a novel deep-learning framework designed to enhance the accuracy of depression evaluation by leveraging both local and global features from video, audio, and text cues.

Methods

The IMDD-Net integrates these multimodal data streams using the Kronecker product for multimodal fusion, facilitating deep interactions between modalities. Within the audio modality, Mel Frequency Cepstrum Coefficient (MFCC) and extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) features capture local and global acoustic properties, respectively. For video data, the TimeSformer network extracts both fine-grained and broad temporal features, while the text modality utilizes a pre-trained BERT model to obtain comprehensive contextual information. The IMDD-Net’s architecture effectively combines these diverse data types to provide a holistic analysis of depressive symptoms.

Results

Experimental results on the AVEC 2014 dataset demonstrate that the IMDD-Net achieves state-of-the-art performance in predicting Beck Depression Inventory-II (BDI-II) scores, with a Root Mean Square Error (RMSE) of 7.55 and a Mean Absolute Error (MAE) of 5.75. A classification to identify potential depression subjects can achieve an accuracy of 0.79.

Conclusion

These results underscore the robustness and precision of the IMDD-Net, highlighting the importance of integrating local and global features across multiple modalities for accurate depression prediction.

Schlagwörter

Stichwörter der Autor:innen

deep learningdepressioninformation fusionlocal and global featuresmultimedia

Autor:innen und Institutionen

  • College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China.
  • Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China.
  • Department of Psychology, The People's Hospital of Liaoning Province, Shenyang, China.
  • Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ, United States.

Zitieren

APA

Li, Y., Yang, X., Zhao, M., Wang, J., Yao, Y., Qian, W., & Qi, S. (2025). Predicting depression by using a novel deep learning model and video-audio-text multimodal data. Frontiers in Psychiatry, 16. https://doi.org/10.3389/fpsyt.2025.1602650

Vancouver

Li Y, Yang X, Zhao M, Wang J, Yao Y, Qian W, et al. Predicting depression by using a novel deep learning model and video-audio-text multimodal data. Frontiers in Psychiatry. 2025;16. doi:10.3389/fpsyt.2025.1602650

BibTeX
@article{Li2025,
  author = {Li, Yifu and Yang, Xueping and Zhao, Meng and Wang, Jiangtao and Yao, Yudong and Qian, Wei and Qi, Shouliang},
  title = {Predicting depression by using a novel deep learning model and video-audio-text multimodal data},
  journal = {Frontiers in Psychiatry},
  year = {2025},
  volume = {16},
  doi = {10.3389/fpsyt.2025.1602650}
}
Kostenlos starten →