EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Multimodal Emotion Recognition in Conversation Based on Hypergraphs

Jiaze Li, Hongyan Mei, Liyun Jia, Xing Zhang

Electronics · 2023-11-19 · 12 (22) · 4703

DOI: 10.3390/electronics12224703 ↗

At a glance

Journal article in Electronics, published online on 2023-11-19. Written by Jiaze Li and 3 co-authors. Cited 8 times so far (Crossref); 33 references. The full text is freely available (open access, CC BY 4.0). Funded by: Liaoning Education Department Scientific Research Project, General project of Liaoning Provincial Department of Education.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

In recent years, sentiment analysis in conversation has garnered increasing attention due to its widespread applications in areas such as social media analytics, sentiment mining, and electronic healthcare. Existing research primarily focuses on sequence learning and graph-based approaches, yet they overlook the high-order interactions between different modalities and the long-term dependencies within each modality. To address these problems, this paper proposes a novel hypergraph-based method for multimodal emotion recognition in conversation (MER-HGraph). MER-HGraph extracts features from three modalities: acoustic, text, and visual. It treats each modality utterance in a conversation as a node and constructs intra-modal hypergraphs (Intra-HGraph) and inter-modal hypergraphs (Inter-HGraph) using hyperedges. The hypergraphs are then updated using hypergraph convolutional networks. Additionally, to mitigate noise in acoustic data and mitigate the impact of fixed time scales, we introduce a dynamic time window module to capture local-global information from acoustic signals. Extensive experiments on the IEMOCAP and MELD datasets demonstrate that MER-HGraph outperforms existing models in multimodal emotion recognition tasks, leveraging high-order information from multimodal data to enhance recognition capabilities.

Funding

  • Liaoning Education Department Scientific Research Project
  • General project of Liaoning Provincial Department of Education

Cite

APA

Li, J., Mei, H., Jia, L., & Zhang, X. (2023). Multimodal Emotion Recognition in Conversation Based on Hypergraphs. Electronics, 12(22), 4703. https://doi.org/10.3390/electronics12224703

Vancouver

Li J, Mei H, Jia L, Zhang X. Multimodal Emotion Recognition in Conversation Based on Hypergraphs. Electronics. 2023;12(22):4703. doi:10.3390/electronics12224703

BibTeX
@article{Li2023,
  author = {Li, Jiaze and Mei, Hongyan and Jia, Liyun and Zhang, Xing},
  title = {Multimodal Emotion Recognition in Conversation Based on Hypergraphs},
  journal = {Electronics},
  year = {2023},
  volume = {12},
  number = {22},
  pages = {4703},
  doi = {10.3390/electronics12224703}
}
Start for free →