Multi-modal multi-task internet sentiment recognition model based on mid-to-back fusion strategy
PLOS One · 10.11.2026 · 21 (11) · e0362468
DOI: 10.1371/journal.pone.0362468 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, online erschienen am 10.11.2026. Verfasst von Sichen Wu und 1 weiteren Autor:innen. Beteiligte Einrichtung: Changsha University of Science and Technology, Changsha, China. Hauptthemen laut MEDLINE: Internet. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Previous studies of multimodal sentiment recognition have paid limited attention to the discrepancy between unimodal expression and multimodal perception in sentiment labeling, and to the question of where in the network the modalities should be fused. This paper proposes a multimodal multi-task sentiment recognition model based on a mid-to-back fusion strategy. Text, audio and visual inputs are encoded with the pre-trained BERT, Wav2Vec 2.0 and CLIP models, respectively, and refined by convolutional, recurrent and self-attention layers. Each unimodal branch is trained with its own unimodal sentiment label, so that modality-specific representations are optimized before fusion; the three high-level unimodal representations are then integrated by a residual multi-head self-attention module supervised by the multimodal label, and the four tasks are optimized jointly. On the Chinese CH-SIMS dataset the proposed model achieves a three-class accuracy of 70.14% and a binary accuracy of 75.09%, improving on the strongest published multi-task baseline by 1.1 and 3.4 percentage points, respectively; the binary-accuracy gain is statistically significant, with p < 0.01 in Welch’s t-test. Fusion-position ablations show that delaying fusion raises neutral-class accuracy from 28.87% with early fusion to 43.82% with mid-to-back fusion, and the model transfers to the English CMU-MOSI benchmark, on which it reaches a binary accuracy of 86.53%. The model still has room for improvement with respect to dataset diversity, the complexity of real-world applications, non-standard language and cross-cultural sentiment understanding. Overall, the proposed approach improves the effectiveness of multimodal fusion and offers a new perspective for multimodal sentiment recognition.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Changsha University of Science and Technology, Changsha, China.
ORCID-Profile: Sichen Wu
Zitieren
APA
Wu, S., & Lin, R. (2026). Multi-modal multi-task internet sentiment recognition model based on mid-to-back fusion strategy. PLOS One, 21(11), e0362468. https://doi.org/10.1371/journal.pone.0362468
Vancouver
Wu S, Lin R. Multi-modal multi-task internet sentiment recognition model based on mid-to-back fusion strategy. PLOS One. 2026;21(11):e0362468. doi:10.1371/journal.pone.0362468
BibTeX
@article{Wu2026,
author = {Wu, Sichen and Lin, Ruina},
title = {Multi-modal multi-task internet sentiment recognition model based on mid-to-back fusion strategy},
journal = {PLOS One},
year = {2026},
volume = {21},
number = {11},
pages = {e0362468},
doi = {10.1371/journal.pone.0362468}
}
EUVIMED