Attention Bidirectional Gated Fusion Based Multimodal Intent Recognition Under Uncertain Missing Modalities
Information · 28.07.2026 · 17 (8) · 728
DOI: 10.3390/info17080728 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information, online erschienen am 28.07.2026. Verfasst von Ling Shang und 5 weiteren Autor:innen. Die Arbeit nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Key Research and Development Project of Shandong Province, Australian Research Council (ARC) Future Fellowship.
Abstract (Originalsprache)
Currently, uncertain missing modalities pose new challenges to multimodal intent recognition. To tackle this issue, this work proposes an Attention Bidirectional Gated Fusion Based Multimodal Intent Recognition model under Uncertain Missing Modalities (named ABGFMIR). Firstly, ABGFMIR extracts the features of each modality (text, audio, visual) with the LSTM network, respectively. Secondly, ABGFMIR narrows the distances between audio, visual and text modality based on Central Moment Discrepancy (CMD), and then performs multimodal feature fusion through an attention bidirectional gated fusion method. Then, corresponding attention level prompts are generated based on the uncertain missing modalities situations of the current sample. The fused multimodal features are then input into the Transformer encoder and decoder, and the prompts are injected into the keys and values of the multihead self-attention layer to guide the Transformer to focus on the missing modes and dynamically adjust the attention distribution, enhancing the robustness of ABGFMIR to different missing modes. Finally, the features produced by the Transformer are fed into the classification layer for intent recognition. Simultaneously, the pre-trained model (AGNN) that trained with the complete modality is employed in the classification layer to guide the main module of ABGFMIR. Two public benchmark datasets (MIntRec and EMOTyDA) are adopted for performance verification. Compared with the other five baseline models, on the MIntRec dataset, ABGFMIR improved accuracy by an average of 2.68 and improved F1 values by an average of 3.24. On the EMOTyDA dataset, ABGFMIR improved accuracy by an average of 2.28 and improved F1 values by an average of 3.44.
Förderung
- National Natural Science Foundation of China
- Key Research and Development Project of Shandong Province
- Australian Research Council (ARC) Future Fellowship
- Discovery Project
- Yantai Institute of Science and Technology’s Research Project
Zitieren
APA
Shang, L., Liu, Z., Wu, Y., Song, X., Yu, J., & Sheng, Q. Z. (2026). Attention Bidirectional Gated Fusion Based Multimodal Intent Recognition Under Uncertain Missing Modalities. Information, 17(8), 728. https://doi.org/10.3390/info17080728
Vancouver
Shang L, Liu Z, Wu Y, Song X, Yu J, Sheng QZ. Attention Bidirectional Gated Fusion Based Multimodal Intent Recognition Under Uncertain Missing Modalities. Information. 2026;17(8):728. doi:10.3390/info17080728
BibTeX
@article{Shang2026,
author = {Shang, Ling and Liu, Zhizhong and Wu, Yuxuan and Song, Xiaoyu and Yu, Jian and Sheng, Quan Z.},
title = {Attention Bidirectional Gated Fusion Based Multimodal Intent Recognition Under Uncertain Missing Modalities},
journal = {Information},
year = {2026},
volume = {17},
number = {8},
pages = {728},
doi = {10.3390/info17080728}
}
EUVIMED