EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal Emotion Recognition in Conversation Based on Hypergraphs

Jiaze Li, Hongyan Mei, Liyun Jia, Xing Zhang

Electronics · 19.11.2023 · 12 (22) · 4703

DOI: 10.3390/electronics12224703 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 19.11.2023. Verfasst von Jiaze Li und 3 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Liaoning Education Department Scientific Research Project, General project of Liaoning Provincial Department of Education.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, sentiment analysis in conversation has garnered increasing attention due to its widespread applications in areas such as social media analytics, sentiment mining, and electronic healthcare. Existing research primarily focuses on sequence learning and graph-based approaches, yet they overlook the high-order interactions between different modalities and the long-term dependencies within each modality. To address these problems, this paper proposes a novel hypergraph-based method for multimodal emotion recognition in conversation (MER-HGraph). MER-HGraph extracts features from three modalities: acoustic, text, and visual. It treats each modality utterance in a conversation as a node and constructs intra-modal hypergraphs (Intra-HGraph) and inter-modal hypergraphs (Inter-HGraph) using hyperedges. The hypergraphs are then updated using hypergraph convolutional networks. Additionally, to mitigate noise in acoustic data and mitigate the impact of fixed time scales, we introduce a dynamic time window module to capture local-global information from acoustic signals. Extensive experiments on the IEMOCAP and MELD datasets demonstrate that MER-HGraph outperforms existing models in multimodal emotion recognition tasks, leveraging high-order information from multimodal data to enhance recognition capabilities.

Förderung

  • Liaoning Education Department Scientific Research Project
  • General project of Liaoning Provincial Department of Education

Zitieren

APA

Li, J., Mei, H., Jia, L., & Zhang, X. (2023). Multimodal Emotion Recognition in Conversation Based on Hypergraphs. Electronics, 12(22), 4703. https://doi.org/10.3390/electronics12224703

Vancouver

Li J, Mei H, Jia L, Zhang X. Multimodal Emotion Recognition in Conversation Based on Hypergraphs. Electronics. 2023;12(22):4703. doi:10.3390/electronics12224703

BibTeX
@article{Li2023,
  author = {Li, Jiaze and Mei, Hongyan and Jia, Liyun and Zhang, Xing},
  title = {Multimodal Emotion Recognition in Conversation Based on Hypergraphs},
  journal = {Electronics},
  year = {2023},
  volume = {12},
  number = {22},
  pages = {4703},
  doi = {10.3390/electronics12224703}
}
Kostenlos starten →