Improving fMRI-based emotion decoding via attention-based multimodal integration
Results in Engineering · 2026-12 · 32 · 112087
DOI: 10.1016/j.rineng.2026.112087 ↗Kurz gefasst
Fachartikel in der Zeitschrift Results in Engineering, 2026-12. Verfasst von Ji-hyuk Ahn und 3 weiteren Autor:innen. Die Arbeit nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Research Foundation of Korea.
Abstract (Originalsprache)
Decoding emotional states from functional magnetic resonance imaging (fMRI) under naturalistic conditions remains a challenging engineering problem due to the high dimensionality, noise, and temporal complexity of neural signals. In addition, emotional responses are influenced by heterogeneous multimodal contextual information, including language, visual content, and acoustic signals, which are typically not integrated with neural data in conventional decoding frameworks. In this study, we propose an attention-based multimodal framework for robust fMRI valence decoding that integrates neural signals with temporally aligned text, audio, and video representations. The proposed architecture incorporates self-attention for intra-modal temporal modeling, an adaptive attention interaction module for reliability-aware cross-modal integration, and a sentiment-guided attention mechanism to emphasize valence-relevant temporal segments. These components are hierarchically fused to construct a unified multimodal representation for three-class valence classification. Experiments conducted on naturalistic fMRI data demonstrate that the proposed multimodal framework provides preliminary evidence of improved within-subject valence prediction relative to the fMRI-only baseline, achieving an accuracy of 62.6% and an F1-score of 60.6%. A leave-one-subject-out analysis further showed higher cross-subject accuracy for the multimodal model than for the fMRI-only baseline. Ablation studies further support the effectiveness of the proposed attention-based modules in improving decoding performance. The results indicate that multimodal integration may provide a useful contextual framework for fMRI-based emotion decoding under naturalistic conditions, although further validation in larger independent cohorts is required. The proposed framework offers a potentially extensible approach for integrating neural and heterogeneous sensory data, with potential applications in affective computing, brain–computer interfaces, and neuroengineering systems.
Förderung
- National Research Foundation of Korea
Zitieren
APA
Ahn, J., Kim, J., Chung, J., & Kim, H. (2026). Improving fMRI-based emotion decoding via attention-based multimodal integration. Results in Engineering, 32, 112087. https://doi.org/10.1016/j.rineng.2026.112087
Vancouver
Ahn J, Kim J, Chung J, Kim H. Improving fMRI-based emotion decoding via attention-based multimodal integration. Results in Engineering. 2026;32:112087. doi:10.1016/j.rineng.2026.112087
BibTeX
@article{Ahn2026,
author = {Ahn, Ji-hyuk and Kim, Jin-Su and Chung, Jin-Young and Kim, Hyun-Chul},
title = {Improving fMRI-based emotion decoding via attention-based multimodal integration},
journal = {Results in Engineering},
year = {2026},
volume = {32},
pages = {112087},
doi = {10.1016/j.rineng.2026.112087}
}
EUVIMED