EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Does small talk with a medical provider affect ChatGPT’s medical counsel? Performance of ChatGPT on USMLE with and without distractions

Myriam Safrai iD, Amos Azaria iD

PLOS ONE · 2024

DOI: 10.1371/journal.pone.0302217 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS ONE, 2024. Verfasst von Myriam Safrai und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Obstetrics and Gynecology, Chaim Sheba Medical Center (Tel Hashomer), Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israe…. Hauptthemen laut MEDLINE: Physician-Patient Relations. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Science and Technology, Israel.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Efforts are being made to improve the time effectiveness of healthcare providers. Artificial intelligence tools can help transcript and summarize physician-patient encounters and produce medical notes and medical recommendations. However, in addition to medical information, discussion between healthcare and patients includes small talk and other information irrelevant to medical concerns. As Large Language Models (LLMs) are predictive models building their response based on the words in the prompts, there is a risk that small talk and irrelevant information may alter the response and the suggestion given. Therefore, this study aims to investigate the impact of medical data mixed with small talk on the accuracy of medical advice provided by ChatGPT. USMLE step 3 questions were used as a model for relevant medical data. We use both multiple-choice and open-ended questions. First, we gathered small talk sentences from human participants using the Mechanical Turk platform. Second, both sets of USLME questions were arranged in a pattern where each sentence from the original questions was followed by a small talk sentence. ChatGPT 3.5 and 4 were asked to answer both sets of questions with and without the small talk sentences. Finally, a board-certified physician analyzed the answers by ChatGPT and compared them to the formal correct answer. The analysis results demonstrate that the ability of ChatGPT-3.5 to answer correctly was impaired when small talk was added to medical data (66.8% vs. 56.6%; p = 0.025). Specifically, for multiple-choice questions (72.1% vs. 68.9%; p = 0.67) and for the open questions (61.5% vs. 44.3%; p = 0.01), respectively. In contrast, small talk phrases did not impair ChatGPT-4 ability in both types of questions (83.6% and 66.2%, respectively). According to these results, ChatGPT-4 seems more accurate than the earlier 3.5 version, and it appears that small talk does not impair its capability to provide medical recommendations. Our results are an important first step in understanding the potential and limitations of utilizing ChatGPT and other LLMs for physician-patient interactions, which include casual conversations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Physician-Patient RelationsHumansFemaleMaleAdultCommunicationHealth PersonnelLicensure, MedicalArtificial IntelligenceCounselingMiddle Aged

Autor:innen und Institutionen

  • Department of Obstetrics and Gynecology, Chaim Sheba Medical Center (Tel Hashomer), Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
  • Department of Obstetrics, Gynecology and Reproductive Sciences, Magee-Womens Research Institute, University of Pittsburgh School of Medicine, Pittsburgh, PA, United States of America.
  • School of Computer Science, Ariel University, Ari'el, Israel.

ORCID-Profile: Myriam Safrai, Amos Azaria

Förderung

  • Ministry of Science and Technology, Israel

Zitieren

APA

Safrai, M., & Azaria, A. (2024). Does small talk with a medical provider affect ChatGPT’s medical counsel? Performance of ChatGPT on USMLE with and without distractions. PLOS ONE. https://doi.org/10.1371/journal.pone.0302217

Vancouver

Safrai M, Azaria A. Does small talk with a medical provider affect ChatGPT’s medical counsel? Performance of ChatGPT on USMLE with and without distractions. PLOS ONE. 2024. doi:10.1371/journal.pone.0302217

BibTeX
@article{Safrai2024,
  author = {Safrai, Myriam and Azaria, Amos},
  title = {Does small talk with a medical provider affect ChatGPT’s medical counsel? Performance of ChatGPT on USMLE with and without distractions},
  journal = {PLOS ONE},
  year = {2024},
  doi = {10.1371/journal.pone.0302217}
}
Kostenlos starten →