Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study
JMIR Medical Education · 10.07.2023 · 9 · e46939
DOI: 10.2196/46939 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Education, online erschienen am 10.07.2023. Verfasst von Oded Nov und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Technology Management, Tandon School of Engineering, New York University, New York, NY, United States. Die Arbeit wurde bisher 176-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Chatbots are being piloted to draft responses to patient questions, but patients’ ability to distinguish between provider and chatbot responses and patients’ trust in chatbots’ functions are not well established.
Objective
This study aimed to assess the feasibility of using ChatGPT (Chat Generative Pre-trained Transformer) or a similar artificial intelligence–based chatbot for patient-provider communication.
Methods
A survey study was conducted in January 2023. Ten representative, nonadministrative patient-provider interactions were extracted from the electronic health record. Patients’ questions were entered into ChatGPT with a request for the chatbot to respond using approximately the same word count as the human provider’s response. In the survey, each patient question was followed by a provider- or ChatGPT-generated response. Participants were informed that 5 responses were provider generated and 5 were chatbot generated. Participants were asked—and incentivized financially—to correctly identify the response source. Participants were also asked about their trust in chatbots’ functions in patient-provider communication, using a Likert scale from 1-5.
Results
A US-representative sample of 430 study participants aged 18 and older were recruited on Prolific, a crowdsourcing platform for academic studies. In all, 426 participants filled out the full survey. After removing participants who spent less than 3 minutes on the survey, 392 respondents remained. Overall, 53.3% (209/392) of respondents analyzed were women, and the average age was 47.1 (range 18-91) years. The correct classification of responses ranged between 49% (192/392) to 85.7% (336/392) for different questions. On average, chatbot responses were identified correctly in 65.5% (1284/1960) of the cases, and human provider responses were identified correctly in 65.1% (1276/1960) of the cases. On average, responses toward patients’ trust in chatbots’ functions were weakly positive (mean Likert score 3.4 out of 5), with lower trust as the health-related complexity of the task in the questions increased.
Conclusions
ChatGPT responses to patient questions were weakly distinguishable from provider responses. Laypeople appear to trust the use of chatbots to answer lower-risk health questions. It is important to continue studying patient-chatbot interaction as chatbots move from administrative to more clinical roles in health care.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Technology Management, Tandon School of Engineering, New York University, New York, NY, United States.
- Department of Population Health, Grossman School of Medicine, New York University, New York, NY, United States.
- Medical Center Information Technology, Langone Health, New York University, New York, NY, United States.
ORCID-Profile: Oded Nov, Nina Singh, Devin Mann
Zitieren
APA
Nov, O., Singh, N., & Mann, D. (2023). Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study. JMIR Medical Education, 9, e46939. https://doi.org/10.2196/46939
Vancouver
Nov O, Singh N, Mann D. Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study. JMIR Medical Education. 2023;9:e46939. doi:10.2196/46939
BibTeX
@article{Nov2023,
author = {Nov, Oded and Singh, Nina and Mann, Devin},
title = {Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study},
journal = {JMIR Medical Education},
year = {2023},
volume = {9},
pages = {e46939},
doi = {10.2196/46939}
}
EUVIMED