EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Cross-modal emotion recognition with causality inference in human conversations

Tahani Jaser Alahmadi, Galiya Ybytayeva, Akbayan Bekarystankyzy, Khalid J. Alzahrani, Rizwan Abbas, Hala Abdelhameed

PeerJ Computer Science · 03.03.2026 · 12 · e3629

DOI: 10.7717/peerj-cs.3629 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PeerJ Computer Science, online erschienen am 03.03.2026. Verfasst von Tahani Jaser Alahmadi und 5 weiteren Autor:innen. Die Arbeit nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Emotion recognition plays an important role in a wide range of application domains. Although previous studies have made progress in this domain, they often fall short in achieving a better understanding of emotions and inferring their underlying causes. To address these limitations, we propose an emotion recognition framework that integrates visual, audio, and textual modalities within a unified architecture. The proposed framework integrates an adaptive cross-modal attention module to capture inter-modal interactions. This module dynamically adjusts the contribution of each modality based on contextual relevance, enhancing recognition accuracy. Additionally, an emotion causality inference module uses a fine-tuned, trainable LLaMA2-Chat (7B) model to jointly process image and text data. This identifies word clues associated with the expressed emotions. Furthermore, a real-time emotion feedback module delivers instantaneous assessments of emotional states during conversations, supporting timely and context-aware interventions. The experimental results on four datasets, SEMAINE, AESI, ECF, and MER-2024, demonstrate that our method achieves improvements in F1-scores compared to baselines.

Förderung

  • Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia

Zitieren

APA

Alahmadi, T. J., Ybytayeva, G., Bekarystankyzy, A., Alzahrani, K. J., Abbas, R., & Abdelhameed, H. (2026). Cross-modal emotion recognition with causality inference in human conversations. PeerJ Computer Science, 12, e3629. https://doi.org/10.7717/peerj-cs.3629

Vancouver

Alahmadi TJ, Ybytayeva G, Bekarystankyzy A, Alzahrani KJ, Abbas R, Abdelhameed H. Cross-modal emotion recognition with causality inference in human conversations. PeerJ Computer Science. 2026;12:e3629. doi:10.7717/peerj-cs.3629

BibTeX
@article{Alahmadi2026,
  author = {Alahmadi, Tahani Jaser and Ybytayeva, Galiya and Bekarystankyzy, Akbayan and Alzahrani, Khalid J. and Abbas, Rizwan and Abdelhameed, Hala},
  title = {Cross-modal emotion recognition with causality inference in human conversations},
  journal = {PeerJ Computer Science},
  year = {2026},
  volume = {12},
  pages = {e3629},
  doi = {10.7717/peerj-cs.3629}
}
Kostenlos starten →