EUVIMEDEUVIMED entdecken
Crossref

Tri-Modal Transformers With Mixture-of-Modality-Experts for Social Media Prediction

Weilong Chen, Wenhao Hu, Xiaolu Chen, Weimin Yuan, Yan Wang, Yanru Zhang, Zhu Han

IEEE Transactions on Circuits and Systems for Video Technology · 2025-02 · 35 (2) · 1897-1909

DOI: 10.1109/tcsvt.2024.3474101 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-02. Verfasst von Weilong Chen und 6 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 67 Literaturangaben. Gefördert von: General Program and Major Project of Shenzhen Science and Technology Research and Development Fund, Project of Chengdu Science and Technology Bureau, Key Research and Development Project of Sichuan Provincial Department of Science and Technology.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With billions of users worldwide, accurately predicting social media popularity is crucial for assessing user behavior, forecasting trends, and enhancing social interactions and business strategies. However, this task presents significant challenges. Firstly, the extraction of valuable insights is complicated by the presence of tri-modal data (visual, text, structured) and pervasive noise. Secondly, the applicability of knowledge acquired during the pre-training phase is often limited due to discrepancies with downstream prediction tasks during the fine-tuning phase. Existing methods for Social Media Popularity Prediction (SMPP), including traditional models and Visual-and-language Models (VLMs), struggle to overcome these challenges, thereby failing to achieve satisfactory accuracy. To tackle these challenges, we propose a novel approach named Tri-Modal Transformers with Mixture-of-Modality-Experts (TTME) for SMPP. TTME integrates Artificial Intelligence Generated Content to mitigate data noise and incorporate a mix of Modality Experts in pre-training phases to effectively utilize tri-modal data. Moreover, to address training disparity, we explore strategies for downstream task adaptation including the integration of diverse pre-training experts and the implementation of DistillSoftmax. Through empirical evaluation, we demonstrate that the TTME significantly improves the accuracy of social media popularity predictions, effectively utilizes tri-modal data with noise, and enhances transferring knowledge from pre-training to downstream tasks.

Förderung

  • General Program and Major Project of Shenzhen Science and Technology Research and Development Fund
  • Project of Chengdu Science and Technology Bureau
  • Key Research and Development Project of Sichuan Provincial Department of Science and Technology
  • Japan Science and Technology Agency (JST) Adopting Sustainable Partnerships for Innovative Research Ecosystem

Zitieren

APA

Chen, W., Hu, W., Chen, X., Yuan, W., Wang, Y., Zhang, Y., & Han, Z. (2025). Tri-Modal Transformers With Mixture-of-Modality-Experts for Social Media Prediction. IEEE Transactions on Circuits and Systems for Video Technology, 35(2), 1897-1909. https://doi.org/10.1109/tcsvt.2024.3474101

Vancouver

Chen W, Hu W, Chen X, Yuan W, Wang Y, Zhang Y, et al. Tri-Modal Transformers With Mixture-of-Modality-Experts for Social Media Prediction. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(2):1897-1909. doi:10.1109/tcsvt.2024.3474101

BibTeX
@article{Chen2025,
  author = {Chen, Weilong and Hu, Wenhao and Chen, Xiaolu and Yuan, Weimin and Wang, Yan and Zhang, Yanru and Han, Zhu},
  title = {Tri-Modal Transformers With Mixture-of-Modality-Experts for Social Media Prediction},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  volume = {35},
  number = {2},
  pages = {1897--1909},
  doi = {10.1109/tcsvt.2024.3474101}
}
Kostenlos starten →