EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Decoding emotions through personalized multi-modal fNIRS-EEG Systems: Exploring deterministic fusion techniques

Alireza F. Nia, Vanessa Tang, Gonzalo D. Maso Talou, Mark Billinghurst

Biomedical Signal Processing and Control · 2025-07 · 105 · 107632

DOI: 10.1016/j.bspc.2025.107632 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Biomedical Signal Processing and Control, 2025-07. Verfasst von Alireza F. Nia und 3 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 84 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: University of Auckland.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Emotion recognition through cortical neurovascular measurements poses significant challenges due to the complex interplay of emotions within cortical activity. Although Electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS) represent a promising combination for developing robust and accurate affective systems, the strategies for their integration remain relatively unexplored. This study aims to address this gap in personalized multimodal fNIRS-EEG emotion recognition systems. Confronted with the scarcity of large-scale multimodal fNIRS and EEG databases for emotion analysis, we developed our own dataset tailored for this research. We assessed widely used conventional models in single-modality emotion recognition to establish a benchmark. Various feature-level and decision-level fusion techniques were then implemented and evaluated against this benchmark. The multimodal fNIRS-EEG system consistently outperformed single-modality systems, achieving notable accuracy improvements of 7.5%, 3%, and 6.5% across the valence, arousal, and dominance dimensions, respectively. These improvements underscore the robustness of multimodal systems in emotion detection. Additionally, the top-performing feature-level fusion technique (direct concatenation) was on par with the best decision-level technique (average-based soft voting) across all dimensions. Our findings confirm the advantages of the multimodal fNIRS-EEG system over single-modality systems emphasizing on both individual characteristics of each modality and complementary nature of their hybrid system and suggest avenues for refining personalized emotion recognition approaches. This highlights the need for further investigation into optimized fusion techniques within alternative emotion recognition pipelines, paving the way for more nuanced and effective methodologies. • Integrating fNIRS and EEG enhances emotion accuracy across key dimensions. • Established a large-scale multimodal fNIRS-EEG dataset for affective analysis. • Analyzed feature and decision-level strategies in neuronal emotion recognition.

Förderung

  • University of Auckland

Zitieren

APA

Nia, A. F., Tang, V., Talou, G. D. M., & Billinghurst, M. (2025). Decoding emotions through personalized multi-modal fNIRS-EEG Systems: Exploring deterministic fusion techniques. Biomedical Signal Processing and Control, 105, 107632. https://doi.org/10.1016/j.bspc.2025.107632

Vancouver

Nia AF, Tang V, Talou GDM, Billinghurst M. Decoding emotions through personalized multi-modal fNIRS-EEG Systems: Exploring deterministic fusion techniques. Biomedical Signal Processing and Control. 2025;105:107632. doi:10.1016/j.bspc.2025.107632

BibTeX
@article{Nia2025,
  author = {Nia, Alireza F. and Tang, Vanessa and Talou, Gonzalo D. Maso and Billinghurst, Mark},
  title = {Decoding emotions through personalized multi-modal fNIRS-EEG Systems: Exploring deterministic fusion techniques},
  journal = {Biomedical Signal Processing and Control},
  year = {2025},
  volume = {105},
  pages = {107632},
  doi = {10.1016/j.bspc.2025.107632}
}
Kostenlos starten →