EUVIMEDEUVIMED entdecken
Qualitative StudieOpen Access

Evaluating Large Language Models for Sentiment Analysis and Hesitancy Analysis on Vaccine Posts From Social Media: Qualitative Study

Augustine Annan iD, Amanda L Eiden iD, Dong Wang iD, Jingcheng Du iD, Majid Rastegar-Mojarad iD, Varun Kumar Nomula iD, Xiaoyan Wang iD

JMIR Formative Research · 15.10.2025 · 9 · e64723-e64723

DOI: 10.2196/64723 ↗

Kurz gefasst

Qualitative Studie in der Zeitschrift JMIR Formative Research, online erschienen am 15.10.2025. Verfasst von Augustine Annan und 6 weiteren Autor:innen. Beteiligte Einrichtung: IMO Health, Rosemont, IL, United States. Hauptthemen laut MEDLINE: Social Media, Vaccination Hesitancy, Language, Vaccines. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

In the digital age, social media has become a crucial platform for public discourse on diverse health-related topics, including vaccines. Efficient sentiment analysis and hesitancy detection are essential for understanding public opinions and concerns. Large language models (LLMs) offer advanced capabilities for processing complex linguistic patterns, potentially providing valuable insights into vaccine-related discourse.

Objective

This study aims to evaluate the performance of various LLMs in sentiment analysis and hesitancy detection related to vaccine discussions on social media and identify the most efficient, accurate, and cost-effective model for detecting vaccine-related public sentiment and hesitancy trends.

Methods

We used several LLMs—generative pretrained transformer (GPT-3.5), GPT-4, Claude-3 Sonnet, and Llama 2—to process and classify complex linguistic data related to human papillomavirus; measles, mumps, and rubella; and vaccines overall from X (formerly known as Twitter), Reddit, and YouTube. The models were tested across different learning paradigms: zero-shot, 1-shot, and few-shot to determine their adaptability and learning efficiency with varying amounts of training data. We evaluated the models’ performance using accuracy, F 1 -score, precision, and recall. In addition, we conducted a cost analysis focused on token usage to assess the computational efficiency of each approach.

Results

GPT-4 ( F 1 -score=0.85 and accuracy=0.83) outperformed GPT-3.5, Llama 2, and Claude-3 Sonnet across various metrics, regardless of the sentiment type or learning paradigm. Few-shot learning did not significantly enhance performance compared with the zero-shot paradigm. Moreover, the increased computational costs and token usage associated with few-shot learning did not justify its application, given the marginal improvement in model performance. The analysis highlighted challenges in classifying neutral sentiments and convenience, correctly interpreting sarcasm, and accurately identifying indirect expressions of vaccine hesitancy, emphasizing the need for model refinement.

Conclusions

GPT-4 emerged as the most accurate model, excelling in sentiment and hesitancy analysis. Performance differences between learning paradigms were minimal, making zero-shot learning preferable for its balance of accuracy and computational efficiency. However, the zero-shot GPT-4 model is not the most cost-effective compared with traditional machine learning. A hybrid approach, using LLMs for initial annotation and traditional models for training, could optimize cost and performance. Despite reliance on specific LLM versions and a limited focus on certain vaccine types and platforms, our findings underscore the capabilities and limitations of LLMs in vaccine sentiment and hesitancy analysis, highlighting the need for ongoing evaluation and adaptation in public health communication strategies.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Social MediaVaccination HesitancyLanguageVaccinesHumansQualitative ResearchLarge Language Models

Stichwörter der Autor:innen

GPT4LLMsNLPartificial intelligencecomputational efficiencyhesitancy analysislanguage modelslarge language modelsmachine learningpublic health communicationpublic opinionpublic sentimentsentiment analysissocial mediasocial media platforms

Substanzen

Vaccines

Autor:innen und Institutionen

  • IMO Health, Rosemont, IL, United States.
  • Merck & Co, Inc, 126 East Lincoln Avenue, Rahway, NJ, 07065, United States, 1 267-305-0672.

ORCID-Profile: Augustine Annan, Amanda L Eiden, Dong Wang, Jingcheng Du, Majid Rastegar-Mojarad, Varun Kumar Nomula, Xiaoyan Wang

Zitieren

APA

Annan, A., Eiden, A. L., Wang, D., Du, J., Rastegar-Mojarad, M., Nomula, V. K., & Wang, X. (2025). Evaluating Large Language Models for Sentiment Analysis and Hesitancy Analysis on Vaccine Posts From Social Media: Qualitative Study. JMIR Formative Research, 9, e64723-e64723. https://doi.org/10.2196/64723

Vancouver

Annan A, Eiden AL, Wang D, Du J, Rastegar-Mojarad M, Nomula VK, et al. Evaluating Large Language Models for Sentiment Analysis and Hesitancy Analysis on Vaccine Posts From Social Media: Qualitative Study. JMIR Formative Research. 2025;9:e64723-e64723. doi:10.2196/64723

BibTeX
@article{Annan2025,
  author = {Annan, Augustine and Eiden, Amanda L and Wang, Dong and Du, Jingcheng and Rastegar-Mojarad, Majid and Nomula, Varun Kumar and Wang, Xiaoyan},
  title = {Evaluating Large Language Models for Sentiment Analysis and Hesitancy Analysis on Vaccine Posts From Social Media: Qualitative Study},
  journal = {JMIR Formative Research},
  year = {2025},
  volume = {9},
  pages = {e64723--e64723},
  doi = {10.2196/64723}
}
Kostenlos starten →