EUVIMEDOpdag EUVIMED
LeitlinieOpen access

Assessing the Adherence of ChatGPT Chatbots to Public Health Guidelines for Smoking Cessation: Content Analysis

Lorien C Abroms iD, Artin Yousefi iD, Christina N Wysota iD, Tien-Chin Wu iD, David A Broniatowski iD

Journal of Medical Internet Research · 2025-01-30 · 27 · e66896

DOI: 10.2196/66896 ↗

At a glance

Leitlinie in Journal of Medical Internet Research, published online on 2025-01-30. Written by Lorien C Abroms and 4 co-authors. Institution: Department of Prevention & Community Health, Milken Institute School of Public Health, George Washington University, Washington, DC, United States. Major MEDLINE topics: Smoking Cessation, Public Health, Guideline Adherence. Cited 26 times so far (Crossref); 18 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Large language model (LLM) artificial intelligence chatbots using generative language can offer smoking cessation information and advice. However, little is known about the reliability of the information provided to users.

Objective

This study aims to examine whether 3 ChatGPT chatbots—the World Health Organization’s Sarah, BeFreeGPT, and BasicGPT—provide reliable information on how to quit smoking.

Methods

A list of quit smoking queries was generated from frequent quit smoking searches on Google related to “how to quit smoking” (n=12). Each query was given to each chatbot, and responses were analyzed for their adherence to an index developed from the US Preventive Services Task Force public health guidelines for quitting smoking and counseling principles. Responses were independently coded by 2 reviewers, and differences were resolved by a third coder.

Results

Across chatbots and queries, on average, chatbot responses were rated as being adherent to 57.1% of the items on the adherence index. Sarah’s adherence (72.2%) was significantly higher than BeFreeGPT (50%) and BasicGPT (47.8%; P

Subjects

MeSH terms (MEDLINE)

Smoking CessationPublic HealthGuideline AdherenceHumansArtificial IntelligenceInternetCounselingGenerative Artificial Intelligence

Author keywords

ChatGPTartificial intelligencechatbotscigaretteslarge language modelssmoking cessationtobacco

Authors and institutions

  • Department of Prevention & Community Health, Milken Institute School of Public Health, George Washington University, Washington, DC, United States.
  • Department of Engineering Management and Systems Engineering, George Washington University, Washington, DC, United States.
  • Department of Population Health, Grossman School of Medicine, New York University, New York, NY, United States.

ORCID profiles: Lorien C Abroms, Artin Yousefi, Christina N Wysota, Tien-Chin Wu, David A Broniatowski

Cite

APA

Abroms, L. C., Yousefi, A., Wysota, C. N., Wu, T. C., & Broniatowski, D. A. (2025). Assessing the Adherence of ChatGPT Chatbots to Public Health Guidelines for Smoking Cessation: Content Analysis. Journal of Medical Internet Research, 27, e66896. https://doi.org/10.2196/66896

Vancouver

Abroms LC, Yousefi A, Wysota CN, Wu TC, Broniatowski DA. Assessing the Adherence of ChatGPT Chatbots to Public Health Guidelines for Smoking Cessation: Content Analysis. Journal of Medical Internet Research. 2025;27:e66896. doi:10.2196/66896

BibTeX
@article{Abroms2025,
  author = {Abroms, Lorien C and Yousefi, Artin and Wysota, Christina N and Wu, Tien-Chin and Broniatowski, David A},
  title = {Assessing the Adherence of ChatGPT Chatbots to Public Health Guidelines for Smoking Cessation: Content Analysis},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e66896},
  doi = {10.2196/66896}
}
Start for free →