Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods
JMIR Formative Research · 2025
DOI: 10.2196/57395 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Lisa M Gandy und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, College of Sciences and Liberal Arts, Kettering University, Flint, MI, United States. Hauptthemen laut MEDLINE: Social Media, Natural Language Processing, Public Health. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Sentiment analysis is one of the most widely used methods for mining and examining text. Social media researchers need guidance on choosing between manual and automated sentiment analysis methods.
Objective
Popular sentiment analysis tools based on natural language processing (NLP; VADER [Valence Aware Dictionary for Sentiment Reasoning], TEXT2DATA [T2D], and Linguistic Inquiry and Word Count [LIWC-22]), and a large language model (ChatGPT 4.0) were compared with manually coded sentiment scores, as applied to the analysis of YouTube comments on videos discussing the opioid epidemic. Sentiment analysis methods were also examined regarding ease of programming, monetary cost, and other practical considerations.
Methods
Evaluation methods included descriptive statistics, receiver operating characteristic (ROC) curve analysis, confusion matrices, Cohen κ, accuracy, specificity, precision, sensitivity (recall), F1-score harmonic mean, and the Matthews correlation coefficient. An inductive, iterative approach to content analysis of the data was used to obtain manual sentiment codes.
Results
A subset of comments were analyzed by a second coder, producing good agreement between the 2 coders’ judgments (κ=0.734). YouTube social media about the opioid crisis had many more negative comments (4286/4871, 88%) than positive comments (79/662, 12%), making it possible to evaluate the performance of sentiment analysis models in an unbalanced dataset. The tone summary measure from LIWC-22 performed better than other tools for estimating the prevalence of negative versus positive sentiment. According to the ROC curve analysis, VADER was best at classifying manually coded negative comments. A comparison of Cohen κ values indicated that NLP tools (VADER, followed by LIWC’s tone and T2D) showed only fair agreement with manual coding. In contrast, ChatGPT 4.0 had poor agreement and failed to generate binary sentiment scores in 2 out of 3 attempts. Variations in accuracy, specificity, precision, sensitivity, F1-score, and MCC did not reveal a single superior model. F1-score harmonic means were 0.34-0.38 (SD 0.02) for NLP tools and very low (0.13) for ChatGPT 4.0. None of the MCCs reached a strong correlation level.
Conclusions
Researchers studying negative emotions, public worries, or dissatisfaction with social media face unique challenges in selecting models suitable for unbalanced datasets. We recommend VADER, the only cost-free tool we evaluated, due to its excellent discrimination, which can be further improved when the comments are at least 100 characters long. If estimating the prevalence of negative comments in an unbalanced dataset is important, we recommend the tone summary measure from LIWC-22. Researchers using T2D must know that it may only score some data and, compared with other methods, be more time-consuming and cost-prohibitive. A general-purpose large language model, ChatGPT 4.0, has yet to surpass the performance of NLP models, at least for unbalanced datasets with highly prevalent (7:1) negative comments.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, College of Sciences and Liberal Arts, Kettering University, Flint, MI, United States.
- Department of Health Administration, The College of Health Professions, Central Michigan University, Mt Pleasant, MI, United States.
- Department of Healthcare Management, Northwood University, Midland, MI, United States.
ORCID-Profile: Lisa M Gandy, Lana V Ivanitskaya, Leeza L Bacon, Rodina Bizri-Baryak
Zitieren
APA
Gandy, L. M., Ivanitskaya, L. V., Bacon, L. L., & Bizri-Baryak, R. (2025). Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods. JMIR Formative Research. https://doi.org/10.2196/57395
Vancouver
Gandy LM, Ivanitskaya LV, Bacon LL, Bizri-Baryak R. Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods. JMIR Formative Research. 2025. doi:10.2196/57395
BibTeX
@article{Gandy2025,
author = {Gandy, Lisa M and Ivanitskaya, Lana V and Bacon, Leeza L and Bizri-Baryak, Rodina},
title = {Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods},
journal = {JMIR Formative Research},
year = {2025},
doi = {10.2196/57395}
}
EUVIMED