EUVIMEDScopri EUVIMED
Open access · CC BY 4.0

Audio-Sensitive Speech Emotion Recognition via Content- Independent Pretraining and Threshold-Based Fusion

Zhaojie Luo, Huaming Xu, Shuqiong Wu

Electronics · 2026-05-27 · 15 (11) · 2313

DOI: 10.3390/electronics15112313 ↗

At a glance

Journal article in Electronics, published online on 2026-05-27. Written by Zhaojie Luo and 2 co-authors. Cited 2 times so far (Crossref); 51 references. The full text is freely available (open access, CC BY 4.0). Funded by: NSFC, Fundamental Research Funds for the Central Universities.

AbstractDati bibliograficiCiteRelated studies

Abstract (original language)

Speech emotion recognition (SER) has attracted increasing attention in human–computer interaction, mental health monitoring, and multimedia retrieval. However, many existing multimodal SER systems exhibit a strong bias toward the text modality: because utterance-level labels are often easily inferred from lexical content, models tend to under-utilize non-verbal acoustic cues, which can lead to erroneous predictions when crucial emotional information is predominantly conveyed by prosodic and spectral features. To alleviate this imbalance, we propose an audio-sensitive SER framework that explicitly enhances the contribution of the audio modality through a two-step strategy. First, we construct an Audio Sensitive Network (ASN) by pretraining on the parallel Emotional Speech Dataset (ESD), in which identical linguistic content is spoken with different emotions. This setting allows the ASN to learn speech content-independent emotional representations that emphasize paralinguistic information. Second, we introduce a threshold fusion scheme that integrates the ASN with existing SER classifiers. Specifically, we employ the Tree-structured Parzen Estimator (TPE) to optimize label-wise decision thresholds, enabling flexible calibration of the joint prediction space across modalities and models. We conduct experiments on both the IEMOCAP and ESD corpora, comparing multiple baseline classifiers with and without the proposed audio-sensitive enhancement. The results show consistent, albeit moderate, improvements in emotion recognition performance (e.g., up to +11.7% absolute accuracy on angry for MMAN on IEMOCAP), particularly for emotions that rely heavily on prosodic and spectral cues, thereby demonstrating the effectiveness of the proposed framework in boosting audio sensitivity within multimodal SER systems.

Funding

  • NSFC
  • Fundamental Research Funds for the Central Universities

Cite

APA

Luo, Z., Xu, H., & Wu, S. (2026). Audio-Sensitive Speech Emotion Recognition via Content- Independent Pretraining and Threshold-Based Fusion. Electronics, 15(11), 2313. https://doi.org/10.3390/electronics15112313

Vancouver

Luo Z, Xu H, Wu S. Audio-Sensitive Speech Emotion Recognition via Content- Independent Pretraining and Threshold-Based Fusion. Electronics. 2026;15(11):2313. doi:10.3390/electronics15112313

BibTeX
@article{Luo2026,
  author = {Luo, Zhaojie and Xu, Huaming and Wu, Shuqiong},
  title = {Audio-Sensitive Speech Emotion Recognition via Content- Independent Pretraining and Threshold-Based Fusion},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {11},
  pages = {2313},
  doi = {10.3390/electronics15112313}
}
Start for free →