EUVIMEDEUVIMED entdecken
Open Access

Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods

Lisa M Gandy iD, Lana V Ivanitskaya iD, Leeza L Bacon iD, Rodina Bizri-Baryak iD

JMIR Formative Research · 2025

DOI: 10.2196/57395 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Lisa M Gandy und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, College of Sciences and Liberal Arts, Kettering University, Flint, MI, United States. Hauptthemen laut MEDLINE: Social Media, Natural Language Processing, Public Health. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Sentiment analysis is one of the most widely used methods for mining and examining text. Social media researchers need guidance on choosing between manual and automated sentiment analysis methods.

Objective

Popular sentiment analysis tools based on natural language processing (NLP; VADER [Valence Aware Dictionary for Sentiment Reasoning], TEXT2DATA [T2D], and Linguistic Inquiry and Word Count [LIWC-22]), and a large language model (ChatGPT 4.0) were compared with manually coded sentiment scores, as applied to the analysis of YouTube comments on videos discussing the opioid epidemic. Sentiment analysis methods were also examined regarding ease of programming, monetary cost, and other practical considerations.

Methods

Evaluation methods included descriptive statistics, receiver operating characteristic (ROC) curve analysis, confusion matrices, Cohen κ, accuracy, specificity, precision, sensitivity (recall), F1-score harmonic mean, and the Matthews correlation coefficient. An inductive, iterative approach to content analysis of the data was used to obtain manual sentiment codes.

Results

A subset of comments were analyzed by a second coder, producing good agreement between the 2 coders’ judgments (κ=0.734). YouTube social media about the opioid crisis had many more negative comments (4286/4871, 88%) than positive comments (79/662, 12%), making it possible to evaluate the performance of sentiment analysis models in an unbalanced dataset. The tone summary measure from LIWC-22 performed better than other tools for estimating the prevalence of negative versus positive sentiment. According to the ROC curve analysis, VADER was best at classifying manually coded negative comments. A comparison of Cohen κ values indicated that NLP tools (VADER, followed by LIWC’s tone and T2D) showed only fair agreement with manual coding. In contrast, ChatGPT 4.0 had poor agreement and failed to generate binary sentiment scores in 2 out of 3 attempts. Variations in accuracy, specificity, precision, sensitivity, F1-score, and MCC did not reveal a single superior model. F1-score harmonic means were 0.34-0.38 (SD 0.02) for NLP tools and very low (0.13) for ChatGPT 4.0. None of the MCCs reached a strong correlation level.

Conclusions

Researchers studying negative emotions, public worries, or dissatisfaction with social media face unique challenges in selecting models suitable for unbalanced datasets. We recommend VADER, the only cost-free tool we evaluated, due to its excellent discrimination, which can be further improved when the comments are at least 100 characters long. If estimating the prevalence of negative comments in an unbalanced dataset is important, we recommend the tone summary measure from LIWC-22. Researchers using T2D must know that it may only score some data and, compared with other methods, be more time-consuming and cost-prohibitive. A general-purpose large language model, ChatGPT 4.0, has yet to surpass the performance of NLP models, at least for unbalanced datasets with highly prevalent (7:1) negative comments.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Social MediaNatural Language ProcessingPublic HealthHumansData MiningROC Curve

Stichwörter der Autor:innen

ChatGPTLIWC-22VADERYouTubedrugsmachine learningmedicationsopioidspharmaceuticalspharmacotherapypopulation healthpublic healthsentiment analysissocial mediavalence aware dictionary for sentiment reasoning

Autor:innen und Institutionen

  • Department of Computer Science, College of Sciences and Liberal Arts, Kettering University, Flint, MI, United States.
  • Department of Health Administration, The College of Health Professions, Central Michigan University, Mt Pleasant, MI, United States.
  • Department of Healthcare Management, Northwood University, Midland, MI, United States.

ORCID-Profile: Lisa M Gandy, Lana V Ivanitskaya, Leeza L Bacon, Rodina Bizri-Baryak

Zitieren

APA

Gandy, L. M., Ivanitskaya, L. V., Bacon, L. L., & Bizri-Baryak, R. (2025). Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods. JMIR Formative Research. https://doi.org/10.2196/57395

Vancouver

Gandy LM, Ivanitskaya LV, Bacon LL, Bizri-Baryak R. Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods. JMIR Formative Research. 2025. doi:10.2196/57395

BibTeX
@article{Gandy2025,
  author = {Gandy, Lisa M and Ivanitskaya, Lana V and Bacon, Leeza L and Bizri-Baryak, Rodina},
  title = {Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/57395}
}
Kostenlos starten →