EUVIMEDSkopri EUVIMED
Open access

Virtual Patients Using Large Language Models: Scalable, Contextualized Simulation of Clinician-Patient Dialogue With Feedback

Virtuelle Patient:innen mit Large Language Models: skalierbare, kontextualisierte Simulation von Clinician-Patient-Dialogen mit Feedback

David A Cook iD, Joshua Overgaard iD, V Shane Pankratz iD, Guilherme Del Fiol iD, Chris A Aakre iD

Journal of Medical Internet Research · 2025-04-04 · 27 · e68486

DOI: 10.2196/68486 ↗

Auf einen Blick

In 60 virtuellen Patient:innen-Dialogen erzeugten LLMs authentische Gespräche und personalisiertes Feedback. GPT-4.0-Turbo schnitt bei Dialogen besser ab als GPT-3.5-Turbo.

Fragestellung

Können mit LLMs angetriebene virtuelle Patient:innen authentische Dialoge erzeugen, Patient:innenpräferenzen abbilden und personalisiertes Feedback zur klinischen Leistung geben?

Methode

Intrinsic evaluation von 60 VP-Clinician-Dialogen zu chronischem Husten und Diabetes; 48 mit menschlichen Clinician, 12 Self-chat; Ratings durch 3 Ärzt:innen und GPT.

Ergebnisse

Mittlere Ratings (maximal 6): Dialogauthentizität 4,7 (0,7), User Experience 4,9 (0,7), Feedbackqualität 4,7 (0,6). Kosten pro Gespräch: US $0,51 mit GPT-4.0-Turbo und US $0,02 mit GPT-3.5-Turbo.

Einschränkungen

Reproduzierbarkeit suboptimal, vor allem wegen variabler LLM-Leistung; keine Bias beobachtet.

Schlussfolgerung der Autor:innen

LLM-gestützte virtuelle Patient:innen können Dialoge simulieren, Patient:innenpräferenzen darstellen und personalisiertes Leistungsfeedback geben. GPT-generierte Feedbackratings ähnelten den menschlichen Ratings.

Kernaussagen

  • Dialogauthentizität im Mittel 4,7/6; User Experience 4,9/6
  • GPT-4.0-Turbo erzielte höhere Dialogratings als GPT-3.5-Turbo
  • Kosten pro Gespräch: US $0,51 bzw. US $0,02

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Journal article in Journal of Medical Internet Research, published online on 2025-04-04. Written by David A Cook and 4 co-authors. Institution: Division of General Internal Medicine, Mayo Clinic College of Medicine and Science, Rochester, MN, United States. Major MEDLINE topics: Physician-Patient Relations, Feedback, Communication, User-Computer Interface. Cited 115 times so far (Crossref); 67 references. The full text is freely available in PubMed Central.

Auf einen BlickAstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Virtual patients (VPs) are computer screen–based simulations of patient-clinician encounters. VP use is limited by cost and low scalability.

Objective

We aimed to show that VPs powered by large language models (LLMs) can generate authentic dialogues, accurately represent patient preferences, and provide personalized feedback on clinical performance. We also explored using LLMs to rate the quality of dialogues and feedback.

Methods

We conducted an intrinsic evaluation study rating 60 VP-clinician conversations. We used carefully engineered prompts to direct OpenAI’s generative pretrained transformer (GPT) to emulate a patient and provide feedback. Using 2 outpatient medicine topics (chronic cough diagnosis and diabetes management), each with permutations representing different patient preferences, we created 60 conversations (dialogues plus feedback): 48 with a human clinician and 12 “self-chat” dialogues with GPT role-playing both the VP and clinician. Primary outcomes were dialogue authenticity and feedback quality, rated using novel instruments for which we conducted a validation study collecting evidence of content, internal structure (reproducibility), relations with other variables, and response process. Each conversation was rated by 3 physicians and by GPT. Secondary outcomes included user experience, bias, patient preferences represented in the dialogues, and conversation features that influenced authenticity.

Results

The average cost per conversation was US $0.51 for GPT-4.0-Turbo and US $0.02 for GPT-3.5-Turbo. Mean (SD) conversation ratings, maximum 6, were overall dialogue authenticity 4.7 (0.7), overall user experience 4.9 (0.7), and average feedback quality 4.7 (0.6). For dialogues created using GPT-4.0-Turbo, physician ratings of patient preferences aligned with intended preferences in 20 to 47 of 48 dialogues (42%-98%). Subgroup comparisons revealed higher ratings for dialogues using GPT-4.0-Turbo versus GPT-3.5-Turbo and for human-generated versus self-chat dialogues. Feedback ratings were similar for human-generated versus GPT-generated ratings, whereas authenticity ratings were lower. We did not perceive bias in any conversation. Dialogue features that detracted from authenticity included that GPT was verbose or used atypical vocabulary (93/180, 51.7% of conversations), was overly agreeable (n=56, 31%), repeated the question as part of the response (n=47, 26%), was easily convinced by clinician suggestions (n=35, 19%), or was not disaffected by poor clinician performance (n=32, 18%). For feedback, detractors included excessively positive feedback (n=42, 23%), failure to mention important weaknesses or strengths (n=41, 23%), or factual inaccuracies (n=39, 22%). Regarding validation of dialogue and feedback scores, items were meticulously developed (content evidence), and we confirmed expected relations with other variables (higher ratings for advanced LLMs and human-generated dialogues). Reproducibility was suboptimal, due largely to variation in LLM performance rather than rater idiosyncrasies.

Conclusions

LLM-powered VPs can simulate patient-clinician dialogues, demonstrably represent patient preferences, and provide personalized performance feedback. This approach is scalable, globally accessible, and inexpensive. LLM-generated ratings of feedback quality are similar to human ratings.

Subjects

MeSH terms (MEDLINE)

Physician-Patient RelationsFeedbackCommunicationUser-Computer InterfacePatient SimulationHumansPatient PreferenceDiabetes MellitusLarge Language Models

Author keywords

clinical decision-makingclinical reasoningcomputer-assisted instructionmachine learningnatural language generationnatural language processingsimulation trainingvirtual patient

Authors and institutions

  • Division of General Internal Medicine, Mayo Clinic College of Medicine and Science, Rochester, MN, United States.
  • Multidisciplinary Simulation Center, Mayo Clinic College of Medicine and Science, Rochester, MN, United States.
  • Health Sciences Center, University of New Mexico, Albuquerque, NM, United States.
  • Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, UT, United States.

ORCID profiles: David A Cook, Joshua Overgaard, V Shane Pankratz, Guilherme Del Fiol, Chris A Aakre

Cite

APA

Cook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G., & Aakre, C. A. (2025). Virtual Patients Using Large Language Models: Scalable, Contextualized Simulation of Clinician-Patient Dialogue With Feedback. Journal of Medical Internet Research, 27, e68486. https://doi.org/10.2196/68486

Vancouver

Cook DA, Overgaard J, Pankratz VS, Del Fiol G, Aakre CA. Virtual Patients Using Large Language Models: Scalable, Contextualized Simulation of Clinician-Patient Dialogue With Feedback. Journal of Medical Internet Research. 2025;27:e68486. doi:10.2196/68486

BibTeX
@article{Cook2025,
  author = {Cook, David A and Overgaard, Joshua and Pankratz, V Shane and Del Fiol, Guilherme and Aakre, Chris A},
  title = {Virtual Patients Using Large Language Models: Scalable, Contextualized Simulation of Clinician-Patient Dialogue With Feedback},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e68486},
  doi = {10.2196/68486}
}
Start for free →