Tri-Modal Transformers With Mixture-of-Modality-Experts for Social Media Prediction
IEEE Transactions on Circuits and Systems for Video Technology · 2025-02 · 35 (2) · 1897-1909
DOI: 10.1109/tcsvt.2024.3474101 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-02. Verfasst von Weilong Chen und 6 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 67 Literaturangaben. Gefördert von: General Program and Major Project of Shenzhen Science and Technology Research and Development Fund, Project of Chengdu Science and Technology Bureau, Key Research and Development Project of Sichuan Provincial Department of Science and Technology.
Abstract (Originalsprache)
With billions of users worldwide, accurately predicting social media popularity is crucial for assessing user behavior, forecasting trends, and enhancing social interactions and business strategies. However, this task presents significant challenges. Firstly, the extraction of valuable insights is complicated by the presence of tri-modal data (visual, text, structured) and pervasive noise. Secondly, the applicability of knowledge acquired during the pre-training phase is often limited due to discrepancies with downstream prediction tasks during the fine-tuning phase. Existing methods for Social Media Popularity Prediction (SMPP), including traditional models and Visual-and-language Models (VLMs), struggle to overcome these challenges, thereby failing to achieve satisfactory accuracy. To tackle these challenges, we propose a novel approach named Tri-Modal Transformers with Mixture-of-Modality-Experts (TTME) for SMPP. TTME integrates Artificial Intelligence Generated Content to mitigate data noise and incorporate a mix of Modality Experts in pre-training phases to effectively utilize tri-modal data. Moreover, to address training disparity, we explore strategies for downstream task adaptation including the integration of diverse pre-training experts and the implementation of DistillSoftmax. Through empirical evaluation, we demonstrate that the TTME significantly improves the accuracy of social media popularity predictions, effectively utilizes tri-modal data with noise, and enhances transferring knowledge from pre-training to downstream tasks.
Förderung
- General Program and Major Project of Shenzhen Science and Technology Research and Development Fund
- Project of Chengdu Science and Technology Bureau
- Key Research and Development Project of Sichuan Provincial Department of Science and Technology
- Japan Science and Technology Agency (JST) Adopting Sustainable Partnerships for Innovative Research Ecosystem
Zitieren
APA
Chen, W., Hu, W., Chen, X., Yuan, W., Wang, Y., Zhang, Y., & Han, Z. (2025). Tri-Modal Transformers With Mixture-of-Modality-Experts for Social Media Prediction. IEEE Transactions on Circuits and Systems for Video Technology, 35(2), 1897-1909. https://doi.org/10.1109/tcsvt.2024.3474101
Vancouver
Chen W, Hu W, Chen X, Yuan W, Wang Y, Zhang Y, et al. Tri-Modal Transformers With Mixture-of-Modality-Experts for Social Media Prediction. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(2):1897-1909. doi:10.1109/tcsvt.2024.3474101
BibTeX
@article{Chen2025,
author = {Chen, Weilong and Hu, Wenhao and Chen, Xiaolu and Yuan, Weimin and Wang, Yan and Zhang, Yanru and Han, Zhu},
title = {Tri-Modal Transformers With Mixture-of-Modality-Experts for Social Media Prediction},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2025},
volume = {35},
number = {2},
pages = {1897--1909},
doi = {10.1109/tcsvt.2024.3474101}
}
EUVIMED