HMDF-Net: a transfer learning-based heterogeneous multimodal dynamic fusion network for depression detection among inmates in correctional facilities

Vollständiger Abstract

Worum geht es in dieser Arbeit?

Objective In correctional institutions, inmates frequently suffer from mental health issues (such as depression), which can easily lead to emergencies like suicide, self-harm, and violent conflicts, posing severe challenges to security management and recidivism prevention. Traditional assessment methods have limitations such as subjectivity, lag, and low efficiency, which may result in misjudgment, omission, and delayed judgment. To address these issues, this study aims to develop an automated depression screening model suitable for the complex scenarios in correctional institutions to achieve objective, accurate, dynamic, and efficient non-contact mental health risk monitoring and early warning. Methods A Heterogeneous Multimodal Dynamic Fusion Network (HMDF-Net) is proposed. This method first constructs a multimodal dataset for inmates and adopts a transfer learning strategy to alleviate the problem of data scarcity. HMDF-Net integrates VGG19, wav2vec 2.0, and BERT to extract visual (facial micro-expressions), audio (speech prosody), and text (dialogue semantics) features respectively. A temporal attention dynamic fusion mechanism is introduced to dynamically assign weights according to the real-time signal quality of each modal channel. Results HMDF-Net achieves an accuracy of 87.5% and a recall rate of 100% on the test set, significantly outperforming various comparison methods. The analysis of the dynamic attention mechanism reveals an intelligent hierarchical fusion paradigm of “visual as the main (48%), text as the auxiliary (32%), and audio as the supplement (20%)”, verifying the core role of the visual modality in combating emotional concealment and the decision-making logic of the multimodal dynamic fusion model in complex environments. Conclusion We present HMDF-Net, a cross-modal fusion network equipped with temporal attention that adjusts feature weights according to signal quality to suppress low-quality multimodal inputs. This framework offers a feasible option for scalable non-invasive mental health screening within correctional settings. Our observations suggest dataset limitations unique to this field matter more to model performance than available computing resources.