EUVIMEDOpdag EUVIMED
Video/Audio

Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing

Lei Zha, YuXue Feng, Zhe Liang

Journal of Visualized Experiments · 2026

DOI: 10.3791/70600 ↗

At a glance

Video/Audio in Journal of Visualized Experiments, 2026. Written by Lei Zha and 2 co-authors. Institution: College of Art, Minzu Normal University of Xingyi; Department of Electrical, Electronic and Automatic Engineering, University of Girona; u6107232@cam…. Major MEDLINE topics: Electroencephalography, Emotions, Facial Expression, Paintings. 22 references.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Emotion recognition during painting viewing remains difficult because aesthetic responses are dynamic, context-dependent, and only partly observable through outward behavior. Unimodal approaches based only on facial expressions or only on physiological signals, therefore, often provide incomplete representations of the viewer's experience. This article describes a reproducible method for constructing a dual-branch cross-attention network that integrates facial video with electroencephalography (EEG) data for multimodal emotion recognition in a painting exhibition context. The protocol begins with the establishment of a synchronized acquisition environment using a 64-channel EEG system and a high-resolution camera for concurrent recording during painting viewing. The subsequent procedure details signal preprocessing, including EEG filtering, segmentation, and differential entropy feature extraction, together with facial detection, alignment, and frame preparation for video analysis. The neural network contains two modality-specific branches. The EEG branch uses a convolutional neural network and a bidirectional long short-term memory network to model spatial-temporal neural features, whereas the facial branch uses a coordinate attention-enhanced lightweight convolutional network to extract visual expression features. A multi-head cross-attention module is then used to fuse the two streams by learning intermodal relevance. Under the experimental conditions reported here, the multimodal framework achieved higher classification accuracy than the unimodal comparison models in four-category emotion recognition. This method is most suitable for offline analysis in controlled acquisition settings and provides a practical framework for affective computing, empirical aesthetics, and exhibition-oriented human-computer interaction research.

Subjects

MeSH terms (MEDLINE)

ElectroencephalographyEmotionsFacial ExpressionPaintingsNeural Networks, ComputerHumansConvolutional Neural Networks

Authors and institutions

  • College of Art, Minzu Normal University of Xingyi; Department of Electrical, Electronic and Automatic Engineering, University of Girona; u6107232@campus.udg.edu.
  • College of Sericulture, Textile and Biomass Sciences, Southwest University.
  • College of Art and Design, Zhengzhou University of Light Industry; Faculty of Education and Psychology, University of Girona.

Cite

APA

Zha, L., Feng, Y., & Liang, Z. (2026). Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing. Journal of Visualized Experiments. https://doi.org/10.3791/70600

Vancouver

Zha L, Feng Y, Liang Z. Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing. Journal of Visualized Experiments. 2026. doi:10.3791/70600

BibTeX
@article{Zha2026,
  author = {Zha, Lei and Feng, YuXue and Liang, Zhe},
  title = {Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing},
  journal = {Journal of Visualized Experiments},
  year = {2026},
  doi = {10.3791/70600}
}
Start for free →