EUVIMEDEUVIMED entdecken
Video/Audio

Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing

Lei Zha, YuXue Feng, Zhe Liang

Journal of Visualized Experiments · 2026

DOI: 10.3791/70600 ↗

Kurz gefasst

Video/Audio in der Zeitschrift Journal of Visualized Experiments, 2026. Verfasst von Lei Zha und 2 weiteren Autor:innen. Beteiligte Einrichtung: College of Art, Minzu Normal University of Xingyi; Department of Electrical, Electronic and Automatic Engineering, University of Girona; u6107232@cam…. Hauptthemen laut MEDLINE: Electroencephalography, Emotions, Facial Expression, Paintings. Die Arbeit nennt 22 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Emotion recognition during painting viewing remains difficult because aesthetic responses are dynamic, context-dependent, and only partly observable through outward behavior. Unimodal approaches based only on facial expressions or only on physiological signals, therefore, often provide incomplete representations of the viewer's experience. This article describes a reproducible method for constructing a dual-branch cross-attention network that integrates facial video with electroencephalography (EEG) data for multimodal emotion recognition in a painting exhibition context. The protocol begins with the establishment of a synchronized acquisition environment using a 64-channel EEG system and a high-resolution camera for concurrent recording during painting viewing. The subsequent procedure details signal preprocessing, including EEG filtering, segmentation, and differential entropy feature extraction, together with facial detection, alignment, and frame preparation for video analysis. The neural network contains two modality-specific branches. The EEG branch uses a convolutional neural network and a bidirectional long short-term memory network to model spatial-temporal neural features, whereas the facial branch uses a coordinate attention-enhanced lightweight convolutional network to extract visual expression features. A multi-head cross-attention module is then used to fuse the two streams by learning intermodal relevance. Under the experimental conditions reported here, the multimodal framework achieved higher classification accuracy than the unimodal comparison models in four-category emotion recognition. This method is most suitable for offline analysis in controlled acquisition settings and provides a practical framework for affective computing, empirical aesthetics, and exhibition-oriented human-computer interaction research.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

ElectroencephalographyEmotionsFacial ExpressionPaintingsNeural Networks, ComputerHumansConvolutional Neural Networks

Autor:innen und Institutionen

  • College of Art, Minzu Normal University of Xingyi; Department of Electrical, Electronic and Automatic Engineering, University of Girona; u6107232@campus.udg.edu.
  • College of Sericulture, Textile and Biomass Sciences, Southwest University.
  • College of Art and Design, Zhengzhou University of Light Industry; Faculty of Education and Psychology, University of Girona.

Zitieren

APA

Zha, L., Feng, Y., & Liang, Z. (2026). Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing. Journal of Visualized Experiments. https://doi.org/10.3791/70600

Vancouver

Zha L, Feng Y, Liang Z. Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing. Journal of Visualized Experiments. 2026. doi:10.3791/70600

BibTeX
@article{Zha2026,
  author = {Zha, Lei and Feng, YuXue and Liang, Zhe},
  title = {Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing},
  journal = {Journal of Visualized Experiments},
  year = {2026},
  doi = {10.3791/70600}
}
Kostenlos starten →