Bridging Text and Speech for Emotion Understanding: An Explainable Multimodal Transformer Fusion Framework with Unified Audio–Text Attribution
Journal of Intelligence · 03.12.2025 · 13 (12) · 159
DOI: 10.3390/jintelligence13120159 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Intelligence, online erschienen am 03.12.2025. Verfasst von Ashutosh Pandey und 2 weiteren Autor:innen. Beteiligte Einrichtung: Computer Science Engineering Department, Thapar Institute of Engineering and Technology, Patiala 147001, Punjab, India. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Conversational interactions, rich in both linguistic and vocal cues, provide a natural context for studying these processes. In this work, we propose an explainable multimodal transformer framework that integrates textual semantics (via RoBERTa) and acoustic prosody (via WavLM) to advance emotion understanding. By projecting both modalities into a shared latent space, our model captures the complementary contributions of language and speech to affective communication, achieving an 0.83 accuracy value across five emotion categories. Crucially, we embed explainable AI (XAI) techniques including Integrated Gradients and Occlusion to attribute predictions to specific linguistic tokens and prosodic patterns, thereby aligning computational mechanisms with human cognitive processes of emotion perception. Beyond performance gains, this work demonstrates how multimodal AI systems can support transparent, human-centered emotion recognition.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Computer Science Engineering Department, Thapar Institute of Engineering and Technology, Patiala 147001, Punjab, India.
ORCID-Profile: Maninder Kaur
Zitieren
APA
Pandey, A., Singh, J., & Kaur, M. (2025). Bridging Text and Speech for Emotion Understanding: An Explainable Multimodal Transformer Fusion Framework with Unified Audio–Text Attribution. Journal of Intelligence, 13(12), 159. https://doi.org/10.3390/jintelligence13120159
Vancouver
Pandey A, Singh J, Kaur M. Bridging Text and Speech for Emotion Understanding: An Explainable Multimodal Transformer Fusion Framework with Unified Audio–Text Attribution. Journal of Intelligence. 2025;13(12):159. doi:10.3390/jintelligence13120159
BibTeX
@article{Pandey2025,
author = {Pandey, Ashutosh and Singh, Jasmeet and Kaur, Maninder},
title = {Bridging Text and Speech for Emotion Understanding: An Explainable Multimodal Transformer Fusion Framework with Unified Audio–Text Attribution},
journal = {Journal of Intelligence},
year = {2025},
volume = {13},
number = {12},
pages = {159},
doi = {10.3390/jintelligence13120159}
}
EUVIMED