EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Jointly Learning From Unimodal and Multimodal-Rated Labels in Audio-Visual Emotion Recognition

Lucas Goncalves, Huang-Cheng Chou, Ali N. Salman, Chi-Chun Lee, Carlos Busso

IEEE Open Journal of Signal Processing · 2025 · 6 · 165-174

DOI: 10.1109/ojsp.2025.3530274 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Open Journal of Signal Processing, 2025. Verfasst von Lucas Goncalves und 4 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Audio-visual emotion recognition(AVER) has been an important research area inhuman-computer interaction(HCI). Traditionally, audio-visual emotional datasets and corresponding models derive their ground truths from annotations obtained by raters after watching the audio-visual stimuli. This conventional method, however, neglects the nuanced human perception of emotional states, which varies when annotations are made under different emotional stimuli conditions—whether through unimodal or multimodal stimuli. This study investigates the potential for enhanced AVER system performance by integrating diverse levels of annotation stimuli, reflective of varying perceptual evaluations. We propose a two-stage training method to train models with the labels elicited by audio-only, face-only, and audio-visual stimuli. Our approach utilizes different levels of annotation stimuli according to which modality is present within different layers of the model, effectively modeling annotation at the unimodal and multi-modal levels to capture the full scope of emotion perception across unimodal and multimodal contexts. We conduct the experiments and evaluate the models on the CREMA-D emotion database. The proposed methods achieved the best performances in macro-/weighted-F1 scores. Additionally, we measure the model calibration, performance bias, and fairness metrics considering the age, gender, and race of the AVER systems.

Förderung

  • National Science Foundation

Zitieren

APA

Goncalves, L., Chou, H., Salman, A. N., Lee, C., & Busso, C. (2025). Jointly Learning From Unimodal and Multimodal-Rated Labels in Audio-Visual Emotion Recognition. IEEE Open Journal of Signal Processing, 6, 165-174. https://doi.org/10.1109/ojsp.2025.3530274

Vancouver

Goncalves L, Chou H, Salman AN, Lee C, Busso C. Jointly Learning From Unimodal and Multimodal-Rated Labels in Audio-Visual Emotion Recognition. IEEE Open Journal of Signal Processing. 2025;6:165-174. doi:10.1109/ojsp.2025.3530274

BibTeX
@article{Goncalves2025,
  author = {Goncalves, Lucas and Chou, Huang-Cheng and Salman, Ali N. and Lee, Chi-Chun and Busso, Carlos},
  title = {Jointly Learning From Unimodal and Multimodal-Rated Labels in Audio-Visual Emotion Recognition},
  journal = {IEEE Open Journal of Signal Processing},
  year = {2025},
  volume = {6},
  pages = {165--174},
  doi = {10.1109/ojsp.2025.3530274}
}
Kostenlos starten →