EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Bridging Text and Speech for Emotion Understanding: An Explainable Multimodal Transformer Fusion Framework with Unified Audio–Text Attribution

Ashutosh Pandey, Jasmeet Singh, Maninder Kaur iD

Journal of Intelligence · 03.12.2025 · 13 (12) · 159

DOI: 10.3390/jintelligence13120159 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Intelligence, online erschienen am 03.12.2025. Verfasst von Ashutosh Pandey und 2 weiteren Autor:innen. Beteiligte Einrichtung: Computer Science Engineering Department, Thapar Institute of Engineering and Technology, Patiala 147001, Punjab, India. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Conversational interactions, rich in both linguistic and vocal cues, provide a natural context for studying these processes. In this work, we propose an explainable multimodal transformer framework that integrates textual semantics (via RoBERTa) and acoustic prosody (via WavLM) to advance emotion understanding. By projecting both modalities into a shared latent space, our model captures the complementary contributions of language and speech to affective communication, achieving an 0.83 accuracy value across five emotion categories. Crucially, we embed explainable AI (XAI) techniques including Integrated Gradients and Occlusion to attribute predictions to specific linguistic tokens and prosodic patterns, thereby aligning computational mechanisms with human cognitive processes of emotion perception. Beyond performance gains, this work demonstrates how multimodal AI systems can support transparent, human-centered emotion recognition.

Schlagwörter

Stichwörter der Autor:innen

emotionexplainable AImultimodal learningspeechtext fusion

Autor:innen und Institutionen

  • Computer Science Engineering Department, Thapar Institute of Engineering and Technology, Patiala 147001, Punjab, India.

ORCID-Profile: Maninder Kaur

Zitieren

APA

Pandey, A., Singh, J., & Kaur, M. (2025). Bridging Text and Speech for Emotion Understanding: An Explainable Multimodal Transformer Fusion Framework with Unified Audio–Text Attribution. Journal of Intelligence, 13(12), 159. https://doi.org/10.3390/jintelligence13120159

Vancouver

Pandey A, Singh J, Kaur M. Bridging Text and Speech for Emotion Understanding: An Explainable Multimodal Transformer Fusion Framework with Unified Audio–Text Attribution. Journal of Intelligence. 2025;13(12):159. doi:10.3390/jintelligence13120159

BibTeX
@article{Pandey2025,
  author = {Pandey, Ashutosh and Singh, Jasmeet and Kaur, Maninder},
  title = {Bridging Text and Speech for Emotion Understanding: An Explainable Multimodal Transformer Fusion Framework with Unified Audio–Text Attribution},
  journal = {Journal of Intelligence},
  year = {2025},
  volume = {13},
  number = {12},
  pages = {159},
  doi = {10.3390/jintelligence13120159}
}
Kostenlos starten →