EUVIMEDEUVIMED entdecken
Crossref

Voices, Faces, and Feelings: Multi-modal Emotion-Cognition Captioning for Mental Health Understanding

Zhiyuan Zhou, Yanrong Guo, Shijie Hao

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (3) · 2263-2271

DOI: 10.1609/aaai.v40i3.37210 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Zhiyuan Zhou und 2 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Emotional and cognitive factors are essential for understanding mental health disorders. However, existing methods often treat multi-modal data as classification tasks, limiting interpretability especially for emotion and cognition. Although large language models (LLMs) offer opportunities for mental health analysis, they mainly rely on textual semantics and overlook fine-grained emotional and cognitive cues in multi-modal inputs. While some studies incorporate emotional features via transfer learning, their connection to mental health conditions remains implicit. To address these issues, we propose ECMC, a novel task that aims at generating natural language descriptions of emotional and cognitive states from multi-modal data, and producing emotion–cognition profiles that improve both the accuracy and interpretability of mental health assessments. We adopt an encoder–decoder architecture, where modality-specific encoders extract features, which are fused by a dual-stream BridgeNet based on Q-former. Contrastive learning enhances the extraction of emotional and cognitive features. A LLaMA decoder then aligns these features with annotated captions to produce detailed descriptions. Extensive objective and subjective evaluations demonstrate that: 1) ECMC outperforms existing multi-modal LLMs and mental health models in generating emotion–cognition captions; 2) the generated emotion–cognition profiles significantly improve assistive diagnosis and interpretability in mental health analysis.

Zitieren

APA

Zhou, Z., Guo, Y., & Hao, S. (2026). Voices, Faces, and Feelings: Multi-modal Emotion-Cognition Captioning for Mental Health Understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 40(3), 2263-2271. https://doi.org/10.1609/aaai.v40i3.37210

Vancouver

Zhou Z, Guo Y, Hao S. Voices, Faces, and Feelings: Multi-modal Emotion-Cognition Captioning for Mental Health Understanding. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(3):2263-2271. doi:10.1609/aaai.v40i3.37210

BibTeX
@article{Zhou2026,
  author = {Zhou, Zhiyuan and Guo, Yanrong and Hao, Shijie},
  title = {Voices, Faces, and Feelings: Multi-modal Emotion-Cognition Captioning for Mental Health Understanding},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {3},
  pages = {2263--2271},
  doi = {10.1609/aaai.v40i3.37210}
}
Kostenlos starten →