HemoGAT: Heterogeneous multimodal speech emotion recognition with cross-modal transformer and graph attention network
Advances in Electrical and Electronic Engineering · 26.06.2026 · 24 (2)
DOI: 10.15598/aeee.v24i2.250415 ↗Kurz gefasst
Fachartikel in der Zeitschrift Advances in Electrical and Electronic Engineering, online erschienen am 26.06.2026. Verfasst von Nhut Minh NGUYEN und 3 weiteren Autor:innen.
Abstract (Originalsprache)
Multimodal speech emotion recognition (SER) is a promising field, yet effectively fusing diverse information streams remains challenging. Addressing this requires architectures capable of modeling structural relationships across modalities with fine-grained, feature- level interactions. This paper proposes HemoGAT, a novel heterogeneous multimodal SER architecture that integrates a dual-stream architecture with two core mod- ules: a heterogeneous multimodal graph attention net- work (HM-GAT) and a cross-modal transformer (CMT) to address this. The HM-GAT module captures complex structural and contextual dependencies using a hetero- geneous graph constructed from deep embeddings. The CMT module enables precise cross-modal feature fusion through bidirectional cross-attention. This design effec- tively captures both high-level relationships and immedi- ate cross-modal influences. HemoGAT achieves state-of- the-art (SOTA) performance on the IEMOCAP dataset and highly competitive results on the MELD dataset, demonstrating its superiority over existing methods. Extensive ablation studies were conducted to evaluate HemoGAT. We assessed the impact of the Top-K algo- rithm for heterogeneous graph construction and com- pared unimodal and multimodal fusion strategies. We also examined the contributions of the HM-GAT and CMT modules, analyzed the role of the graph attention network (GAT) in graph learning, and evaluated the effect of GAT layer depth on performance
Zitieren
APA
NGUYEN, N. M., NGUYEN, T. T., NGUYEN, T., & DANG, D. N. M. (2026). HemoGAT: Heterogeneous multimodal speech emotion recognition with cross-modal transformer and graph attention network. Advances in Electrical and Electronic Engineering, 24(2). https://doi.org/10.15598/aeee.v24i2.250415
Vancouver
NGUYEN NM, NGUYEN TT, NGUYEN T, DANG DNM. HemoGAT: Heterogeneous multimodal speech emotion recognition with cross-modal transformer and graph attention network. Advances in Electrical and Electronic Engineering. 2026;24(2). doi:10.15598/aeee.v24i2.250415
BibTeX
@article{NGUYEN2026,
author = {NGUYEN, Nhut Minh and NGUYEN, Thanh Trung and NGUYEN, Tien-Dat and DANG, Duc Ngoc Minh},
title = {HemoGAT: Heterogeneous multimodal speech emotion recognition with cross-modal transformer and graph attention network},
journal = {Advances in Electrical and Electronic Engineering},
year = {2026},
volume = {24},
number = {2},
doi = {10.15598/aeee.v24i2.250415}
}
EUVIMED