Audio-visual speech enhancement in noisy environments using emotion-based contextual cues
The Journal of the Acoustical Society of America · 2026
DOI: 10.1121/10.0042239 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Journal of the Acoustical Society of America, 2026. Verfasst von Tassadaq Hussain und 8 weiteren Autor:innen. Beteiligte Einrichtung: School of Computing Engineering and the Built Environment, Edinburgh Napier University, Edinburgh, EH105DT, United Kingdom. Hauptthemen laut MEDLINE: Noise, Emotions, Cues, Speech Perception. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 58 Literaturangaben. Gefördert von: CANDU Owners Group.
Abstract (Originalsprache)
In real-world environments, background noise significantly degrades the intelligibility and clarity of human speech. Existing audio-visual speech enhancement (AVSE) techniques often pose challenges in dynamic and noisy conditions. This study examines the inclusion of emotional features as a novel contextual cue within the AVSE framework. We analyze that incorporating emotional understanding from facial landmarks improves speech enhancement performance. We propose a deep learning–based emotion-aware audio-visual speech enhancement system (EAVSE) that uses auditory, visual, and emotional information. The proposed EAVSE extracts emotional features from facial landmarks and combines them with audio and visual modalities. Enriched multi-model data are processed by a UNet-based encoder–decoder network for joint learning and optimization. The network iteratively refines the feature representation, guided by a distortion-inspired loss function. We train and evaluate the model on the Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity dataset, known for its diverse audio-visual recordings with annotated emotions. Compared to AVSE benchmark and audio-only speech enhancement systems, the proposed model achieves significant improvements in both objective [Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI)] and subjective speech quality metrics. In particular, the scale-invariant signal-to-distortion ratio loss function demonstrates superior performance. This suggests the usefulness of the emotional contextual cues for AVSE. The experimental findings demonstrate the effectiveness of the AVSE, particularly in challenging noisy environments [signal-to-noise ratio (SNR) ≤ −7.5 dB]. The proposed model achieved Δ STOI of 7.32%, Δ PESQ of 0.33, and Δ S-SNR of 7.8 dB over noisy benchmark at 0 dB SNR.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- School of Computing Engineering and the Built Environment, Edinburgh Napier University, Edinburgh, EH105DT, United Kingdom.
- Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan.
- School of Engineering, The University of Edinburgh, Edinburgh, EH9 3JL, United Kingdom.
ORCID-Profile: Nasir Saleem, Yu Tsao
Förderung
- CANDU Owners Group
Zitieren
APA
Hussain, T., Saleem, N., Dashtipour, K., Ahmed, S., Hou, J. C., Anwar, U., Tsao, Y., Arsalan, T., & Hussain, A. (2026). Audio-visual speech enhancement in noisy environments using emotion-based contextual cues. The Journal of the Acoustical Society of America. https://doi.org/10.1121/10.0042239
Vancouver
Hussain T, Saleem N, Dashtipour K, Ahmed S, Hou JC, Anwar U, et al. Audio-visual speech enhancement in noisy environments using emotion-based contextual cues. The Journal of the Acoustical Society of America. 2026. doi:10.1121/10.0042239
BibTeX
@article{Hussain2026,
author = {Hussain, Tassadaq and Saleem, Nasir and Dashtipour, Kia and Ahmed, Shafique and Hou, Jen-Cheng and Anwar, Usman and Tsao, Yu and Arsalan, Tughrul and Hussain, Amir},
title = {Audio-visual speech enhancement in noisy environments using emotion-based contextual cues},
journal = {The Journal of the Acoustical Society of America},
year = {2026},
doi = {10.1121/10.0042239}
}
EUVIMED