Multimodal Emotion Detection in Distributed Data Streams: Deep Transformer Ensembles, Cross-Modal Attention Fusion, and Benchmark Evaluation
International Journal of Creative and Open Research in Engineering and Management · 07.11.2026 · 02 (11) · 1-9
DOI: 10.55041/ijcope.v2i11.028 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Creative and Open Research in Engineering and Management, online erschienen am 07.11.2026. Verfasst von Rahul Shankar Pandey und 3 weiteren Autor:innen.
Abstract (Originalsprache)
Emotion detection represents a vital frontier at the inter-section of affective computing, artificial intelligence, and human-computer interaction (HCI). While traditional uni-modal systems—relying solely on text, speech acoustics, or static facial photographs—have established founda-tional benchmarks, they remain fundamentally fragile in real-world conversational dialogues characterized by lex-ical ambiguity, acoustic sarcasm, and visual occlusions. In this paper, we propose an end-to-end, publication-grade deep learning architecture for Multimodal Emo-tion Detection in Distributed Big Data Streams. Our framework synchronizes three specialized foundation models: a linguistic contextual transformer (RoBERTa-Large), a self-supervised acoustic representation encoder (Wav2Vec 2.0), and a spatiotemporal Vision Transformer (ViT-B/16) integrated with Facial Action Unit (FAU) tracking. To resolve temporal granularity discrepancies and inter-modal information imbalance, we formulate a novel Hierarchical Cross-Modal Attention Fusion (HCAF) network. The HCAF module leverages bidirec-tional multi-head cross-attention to project acoustic and vi-sual features into the linguistic latent subspace, selectively attending to emotional cues across communicative chan-nels. To address severe affective class skew (where neutral utterances dominate dialogue corpora), we design a hy-brid multi-task loss combining Focal Loss (γ = 2.0) with Concordance Correlation Coefficient (CCC) optimization for continuous Valence-Arousal dimensional estimation. Furthermore, we architect a distributed stream ingestion pipeline capable of sustaining over 450 concurrent mul-timodal streams per GPU node with sub-50ms latency, incorporating biometric facial landmark de-identification and vocal tract perturbation to guarantee user privacy under GDPR Article 9. We perform extensive empirical evaluations on the Multimodal EmotionLines Dataset (MELD) and the Interactive Emotional Dyadic Mo-tion Capture (IEMOCAP) corpus. Our HCAF model achieves a weighted F1-score of 71.6% on MELD (7-class) and 82.3% on IEMOCAP (4-class), outperforming uni-modal text, audio, and visual baselines by 7.7%, 27.5%, and 30.8% respectively, while surpassing standard Early and Late fusion paradigms by 5.9% and 4.6%. Ablation experiments and Pareto latency analyses confirm that cross-modal self-attention and dynamic loss re-weighting are crucial for robust, real-time affective computing. Keywords: Affective Computing, Multimodal Emotion Recognition, Vision Transformer, Wav2Vec 2.0, RoBERTa, Cross-Attention Fusion, Focal Loss, Distributed Stream Analytics, Biometric Privacy Preservation.
Zitieren
APA
Pandey, R. S., Niyazi, M. S., Mandal, I., & S, V. (2026). Multimodal Emotion Detection in Distributed Data Streams: Deep Transformer Ensembles, Cross-Modal Attention Fusion, and Benchmark Evaluation. International Journal of Creative and Open Research in Engineering and Management, 02(11), 1-9. https://doi.org/10.55041/ijcope.v2i11.028
Vancouver
Pandey RS, Niyazi MS, Mandal I, S V. Multimodal Emotion Detection in Distributed Data Streams: Deep Transformer Ensembles, Cross-Modal Attention Fusion, and Benchmark Evaluation. International Journal of Creative and Open Research in Engineering and Management. 2026;02(11):1-9. doi:10.55041/ijcope.v2i11.028
BibTeX
@article{Pandey2026,
author = {Pandey, Rahul Shankar and Niyazi, Md. Saddab and Mandal, Indrajeet and S, Varun},
title = {Multimodal Emotion Detection in Distributed Data Streams: Deep Transformer Ensembles, Cross-Modal Attention Fusion, and Benchmark Evaluation},
journal = {International Journal of Creative and Open Research in Engineering and Management},
year = {2026},
volume = {02},
number = {11},
pages = {1--9},
doi = {10.55041/ijcope.v2i11.028}
}
EUVIMED