EUVIMEDEUVIMED entdecken
Open Access

Decoding Digital Discourse Through Multimodal Text and Image Machine Learning Models to Classify Sentiment and Detect Hate Speech in Race- and Lesbian, Gay, Bisexual, Transgender, Queer, Intersex, and Asexual Community–Related Posts on Social Media: Quantitative Study

Thu T Nguyen iD, Xiaohe Yue iD, Heran Mane iD, Kyle Seelman iD, Penchala Sai Priya Mullaputi iD, Elizabeth Dennard iD, Amrutha S Alibilli iD, Junaid S Merchant iD, Shaniece Criss iD, Yulin Hswen iD, Quynh C Nguyen iD

Journal of Medical Internet Research · 2025

DOI: 10.2196/72822 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Thu T Nguyen und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Epidemiology and Biostatistics, University of Maryland, College Park, College Park, MD, United States. Hauptthemen laut MEDLINE: Social Media, Machine Learning, Hate. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 93 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIMHD NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

A major challenge in sentiment analysis on social media is the increasing prevalence of image-based content, which integrates text and visuals to convey nuanced messages. Traditional text-based approaches have been widely used to assess public attitudes and beliefs; however, they often fail to fully capture the meaning of multimodal content where cultural, contextual, and visual elements play a significant role.

Objective

This study aims to provide practical guidance for collecting, processing, and analyzing social media data using multimodal machine learning models. Specifically, it focuses on training and fine-tuning models to classify sentiment and detect hate speech.

Methods

Social media data were collected from Facebook and Instagram using CrowdTangle, a public insights tool by Meta, and from X via its academic research application programming interface. The dataset was filtered to include only race-related terms and lesbian, gay, bisexual, transgender, queer, intersex, and asexual community–related posts with image attachments, ensuring focus on multimodal content. Human annotators labeled 13,000 posts into 4 categories: negative sentiment, positive sentiment, hate, or antihate. We evaluated unimodal (Bidirectional Encoder Representations from Transformers for text and Visual Geometry Group 16 for images) and multimodal (Contrastive Language-Image Pretraining [CLIP], Visual Bidirectional Encoder Representations from Transformers [VisualBERTs], and an intermediate fusion) models. To enhance model performance, the synthetic minority oversampling technique was applied to address class imbalances, and latent Dirichlet allocation was used to improve semantic representations.

Results

Our findings highlighted key differences in model performance. Among unimodal models, Bidirectional Encoder Representations from Transformer outperformed Visual Geometry Group 16, achieving higher accuracy and macro–F1-scores across all tasks. Among multimodal models, CLIP achieved the highest accuracy (0.86) in negative sentiment detection, followed by VisualBERT (0.84). For positive sentiment, VisualBERT outperformed other models with the highest accuracy (0.76). In hate speech detection, the intermediate fusion model demonstrated the highest accuracy (0.91) with a macro–F1-score of 0.64, ensuring balanced performance. Meanwhile, VisualBERT performed best in antihate classification, achieving an accuracy of 0.78. Applying latent Dirichlet allocation and the synthetic minority oversampling technique improved minority class detection, particularly for antihate content. Overall, the intermediate fusion model provided the most balanced performance across tasks, while CLIP excelled in accuracy-driven classifications. Although VisualBERT performed well in certain areas, it struggled to maintain a precision-recall balance. These results emphasized the effectiveness of multimodal approaches over unimodal models in analyzing social media sentiment.

Conclusions

This study contributes to the growing research on multimodal machine learning by demonstrating how advanced models, data augmentation techniques, and diverse datasets can enhance the analysis of social media content. The findings offer valuable insights for researchers, policy makers, and public health professionals seeking to leverage artificial intelligence for social media monitoring and addressing broader societal challenges.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Social MediaMachine LearningHateHumansFemaleMaleSexual and Gender Minorities

Stichwörter der Autor:innen

AIartificial intelligencecultural determinants of healthhate speech detectionmemesmultimodal machine learningpublic healthsentiment analysissocial media analysis

Autor:innen und Institutionen

  • Department of Epidemiology and Biostatistics, University of Maryland, College Park, College Park, MD, United States.
  • Department of Computer Science, University of Maryland, College Park, College Park, MD, United States.
  • Department of Health Sciences, Furman University, Greenville, United States.
  • Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, United States.

ORCID-Profile: Thu T Nguyen, Xiaohe Yue, Heran Mane, Kyle Seelman, Penchala Sai Priya Mullaputi, Elizabeth Dennard, Amrutha S Alibilli, Junaid S Merchant, Shaniece Criss, Yulin Hswen, Quynh C Nguyen

Förderung

  • NIMHD NIH HHS

Zitieren

APA

Nguyen, T. T., Yue, X., Mane, H., Seelman, K., Mullaputi, P. S. P., Dennard, E., Alibilli, A. S., Merchant, J. S., Criss, S., Hswen, Y., & Nguyen, Q. C. (2025). Decoding Digital Discourse Through Multimodal Text and Image Machine Learning Models to Classify Sentiment and Detect Hate Speech in Race- and Lesbian, Gay, Bisexual, Transgender, Queer, Intersex, and Asexual Community–Related Posts on Social Media: Quantitative Study. Journal of Medical Internet Research. https://doi.org/10.2196/72822

Vancouver

Nguyen TT, Yue X, Mane H, Seelman K, Mullaputi PSP, Dennard E, et al. Decoding Digital Discourse Through Multimodal Text and Image Machine Learning Models to Classify Sentiment and Detect Hate Speech in Race- and Lesbian, Gay, Bisexual, Transgender, Queer, Intersex, and Asexual Community–Related Posts on Social Media: Quantitative Study. Journal of Medical Internet Research. 2025. doi:10.2196/72822

BibTeX
@article{Nguyen2025,
  author = {Nguyen, Thu T and Yue, Xiaohe and Mane, Heran and Seelman, Kyle and Mullaputi, Penchala Sai Priya and Dennard, Elizabeth and Alibilli, Amrutha S and Merchant, Junaid S and Criss, Shaniece and Hswen, Yulin and Nguyen, Quynh C},
  title = {Decoding Digital Discourse Through Multimodal Text and Image Machine Learning Models to Classify Sentiment and Detect Hate Speech in Race- and Lesbian, Gay, Bisexual, Transgender, Queer, Intersex, and Asexual Community–Related Posts on Social Media: Quantitative Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/72822}
}
Kostenlos starten →