EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Improving fMRI-based emotion decoding via attention-based multimodal integration

Ji-hyuk Ahn, Jin-Su Kim, Jin-Young Chung, Hyun-Chul Kim

Results in Engineering · 2026-12 · 32 · 112087

DOI: 10.1016/j.rineng.2026.112087 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Results in Engineering, 2026-12. Verfasst von Ji-hyuk Ahn und 3 weiteren Autor:innen. Die Arbeit nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Research Foundation of Korea.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Decoding emotional states from functional magnetic resonance imaging (fMRI) under naturalistic conditions remains a challenging engineering problem due to the high dimensionality, noise, and temporal complexity of neural signals. In addition, emotional responses are influenced by heterogeneous multimodal contextual information, including language, visual content, and acoustic signals, which are typically not integrated with neural data in conventional decoding frameworks. In this study, we propose an attention-based multimodal framework for robust fMRI valence decoding that integrates neural signals with temporally aligned text, audio, and video representations. The proposed architecture incorporates self-attention for intra-modal temporal modeling, an adaptive attention interaction module for reliability-aware cross-modal integration, and a sentiment-guided attention mechanism to emphasize valence-relevant temporal segments. These components are hierarchically fused to construct a unified multimodal representation for three-class valence classification. Experiments conducted on naturalistic fMRI data demonstrate that the proposed multimodal framework provides preliminary evidence of improved within-subject valence prediction relative to the fMRI-only baseline, achieving an accuracy of 62.6% and an F1-score of 60.6%. A leave-one-subject-out analysis further showed higher cross-subject accuracy for the multimodal model than for the fMRI-only baseline. Ablation studies further support the effectiveness of the proposed attention-based modules in improving decoding performance. The results indicate that multimodal integration may provide a useful contextual framework for fMRI-based emotion decoding under naturalistic conditions, although further validation in larger independent cohorts is required. The proposed framework offers a potentially extensible approach for integrating neural and heterogeneous sensory data, with potential applications in affective computing, brain–computer interfaces, and neuroengineering systems.

Förderung

  • National Research Foundation of Korea

Zitieren

APA

Ahn, J., Kim, J., Chung, J., & Kim, H. (2026). Improving fMRI-based emotion decoding via attention-based multimodal integration. Results in Engineering, 32, 112087. https://doi.org/10.1016/j.rineng.2026.112087

Vancouver

Ahn J, Kim J, Chung J, Kim H. Improving fMRI-based emotion decoding via attention-based multimodal integration. Results in Engineering. 2026;32:112087. doi:10.1016/j.rineng.2026.112087

BibTeX
@article{Ahn2026,
  author = {Ahn, Ji-hyuk and Kim, Jin-Su and Chung, Jin-Young and Kim, Hyun-Chul},
  title = {Improving fMRI-based emotion decoding via attention-based multimodal integration},
  journal = {Results in Engineering},
  year = {2026},
  volume = {32},
  pages = {112087},
  doi = {10.1016/j.rineng.2026.112087}
}
Kostenlos starten →