EUVIMEDEUVIMED entdecken
Mixed-Methods-StudieOpen Access

A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study

Friederike Holderried iD, Christian Stegemann-Philipps iD, Lea Herschbach iD, Julia-Astrid Moldt iD, Andrew Nevins iD, Jan Griewatz iD, Martin Holderried iD, Anne Herrmann-Werner iD, Teresa Festl-Wietek iD, Moritz Mahling iD

JMIR Medical Education · 16.01.2024 · 10 · e53961

DOI: 10.2196/53961 ↗

Kurz gefasst

Mixed-Methods-Studie in der Zeitschrift JMIR Medical Education, online erschienen am 16.01.2024. Verfasst von Friederike Holderried und 9 weiteren Autor:innen. Beteiligte Einrichtung: Tübingen Institute for Medical Education, Eberhard Karls University, Tübingen, Germany. Hauptthemen laut MEDLINE: Communication, Students, Medical. Die Arbeit wurde bisher 176-mal zitiert (Crossref) und nennt 56 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions Objective The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication.

Methods

We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ).

Results

A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points).

Conclusions

Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

CommunicationStudents, MedicalHumansYoung AdultAdultProspective StudiesLanguageMedical History Taking

Stichwörter der Autor:innen

ChatGPTGPTNLPansweranswersartificial intelligencechatbotchatbotsconversational agentconversational agentsdocumentationgenerative pretrained transformerhistoryhistory takinghuman computer

Autor:innen und Institutionen

  • Tübingen Institute for Medical Education, Eberhard Karls University, Tübingen, Germany.
  • Division of Infectious Diseases, Stanford University School of Medicine, Stanford, CA, United States.
  • Department of Medical Development, Process and Quality Management, University Hospital Tübingen, Tübingen, Germany.
  • Department of Diabetology, Endocrinology, Nephrology, Section of Nephrology and Hypertension, University Hospital Tübingen, Tübingen, Germany.

ORCID-Profile: Friederike Holderried, Christian Stegemann-Philipps, Lea Herschbach, Julia-Astrid Moldt, Andrew Nevins, Jan Griewatz, Martin Holderried, Anne Herrmann-Werner, Teresa Festl-Wietek, Moritz Mahling

Zitieren

APA

Holderried, F., Stegemann-Philipps, C., Herschbach, L., Moldt, J. A., Nevins, A., Griewatz, J., Holderried, M., Herrmann-Werner, A., Festl-Wietek, T., & Mahling, M. (2024). A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study. JMIR Medical Education, 10, e53961. https://doi.org/10.2196/53961

Vancouver

Holderried F, Stegemann-Philipps C, Herschbach L, Moldt JA, Nevins A, Griewatz J, et al. A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study. JMIR Medical Education. 2024;10:e53961. doi:10.2196/53961

BibTeX
@article{Holderried2024,
  author = {Holderried, Friederike and Stegemann-Philipps, Christian and Herschbach, Lea and Moldt, Julia-Astrid and Nevins, Andrew and Griewatz, Jan and Holderried, Martin and Herrmann-Werner, Anne and Festl-Wietek, Teresa and Mahling, Moritz},
  title = {A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study},
  journal = {JMIR Medical Education},
  year = {2024},
  volume = {10},
  pages = {e53961},
  doi = {10.2196/53961}
}
Kostenlos starten →