EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Synthetic Patient–Physician Conversations Simulated by Large Language Models: A Multi-Dimensional Evaluation

Syed Ali Haider iD, Srinivasagam Prabha, Cesar Abraham Gomez-Cabello, Sahar Borna, Ariana Genovese, Maissa Trabilsy, Bernardo G Collaco iD, Nadia G Wood iD, Sanjay Bagaria, Cui Tao iD, Antonio Jorge Forte iD

Sensors · 2025

DOI: 10.3390/s25144305 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2025. Verfasst von Syed Ali Haider und 10 weiteren Autor:innen. Beteiligte Einrichtung: Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA. Hauptthemen laut MEDLINE: Physician-Patient Relations, Language, Physicians. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Eric Schmidt Fund, Dalio Philanthropies, Gerstner Philanthropies, Mayo Clinic and the generosity of Eric and Wendy Schmidt.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Data accessibility remains a significant barrier in healthcare AI due to privacy constraints and logistical challenges. Synthetic data, which mimics real patient information while remaining both realistic and non-identifiable, offers a promising solution. Large Language Models (LLMs) create new opportunities to generate high-fidelity clinical conversations between patients and physicians. However, the value of this synthetic data depends on careful evaluation of its realism, accuracy, and practical relevance.

Objective

To assess the performance of four leading LLMs: ChatGPT 4.5, ChatGPT 4o, Claude 3.7 Sonnet, and Gemini Pro 2.5 in generating synthetic transcripts of patient–physician interactions in plastic surgery scenarios.

Methods

Each model generated transcripts for ten plastic surgery scenarios. Transcripts were independently evaluated by three clinically trained raters using a seven-criterion rubric: Medical Accuracy, Realism, Persona Consistency, Fidelity, Empathy, Relevancy, and Usability. Raters were blinded to the model identity to reduce bias. Each was rated on a 5-point Likert scale, yielding 840 total evaluations. Descriptive statistics were computed, and a two-way repeated measures ANOVA was used to test for differences across models and metrics. In addition, transcripts were analyzed using automated linguistic and content-based metrics.

Results

All models achieved strong performance, with mean ratings exceeding 4.5 across all criteria. Gemini 2.5 Pro received mean scores (5.00 ± 0.00) in Medical Accuracy, Realism, Persona Consistency, Relevancy, and Usability. Claude 3.7 Sonnet matched the scores in Persona Consistency and Relevancy and led in Empathy (4.96 ± 0.18). ChatGPT 4.5 also achieved perfect scores in Relevancy, with high scores in Empathy (4.93 ± 0.25) and Usability (4.96 ± 0.18). ChatGPT 4o demonstrated consistently strong but slightly lower performance across most dimensions. ANOVA revealed no statistically significant differences across models (F(3, 6) = 0.85, p = 0.52). Automated analysis showed substantial variation in transcript length, style, and content richness: Gemini 2.5 Pro generated the longest and most emotionally expressive dialogues, while ChatGPT 4o produced the shortest and most concise outputs.

Conclusions

Leading LLMs can generate medically accurate, emotionally appropriate synthetic dialogues suitable for educational and research use. Despite high performance, demographic homogeneity in generated patients highlights the need for improved diversity and bias mitigation in model outputs. These findings support the cautious, context-aware integration of LLM-generated dialogues into medical training, simulation, and research.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Physician-Patient RelationsLanguagePhysiciansHumansCommunicationLarge Language Models

Stichwörter der Autor:innen

clinical communicationlarge language modelsmedical consultationsmedical simulationpatient–physician interactionsynthetic data

Autor:innen und Institutionen

  • Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA.
  • Department of Radiology AI IT, Mayo Clinic, Rochester, MN 55905, USA.
  • Division of Surgical Oncology, Mayo Clinic, Jacksonville, FL 32224, USA.
  • Department of Artificial Intelligence and Informatics, Mayo Clinic, Jacksonville, FL 32224, USA.
  • Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA.

ORCID-Profile: Syed Ali Haider, Bernardo G Collaco, Nadia G Wood, Cui Tao, Antonio Jorge Forte

Förderung

  • Eric Schmidt Fund, Dalio Philanthropies, Gerstner Philanthropies
  • Mayo Clinic and the generosity of Eric and Wendy Schmidt

Zitieren

APA

Haider, S. A., Prabha, S., Gomez-Cabello, C. A., Borna, S., Genovese, A., Trabilsy, M., Collaco, B. G., Wood, N. G., Bagaria, S., Tao, C., & Forte, A. J. (2025). Synthetic Patient–Physician Conversations Simulated by Large Language Models: A Multi-Dimensional Evaluation. Sensors. https://doi.org/10.3390/s25144305

Vancouver

Haider SA, Prabha S, Gomez-Cabello CA, Borna S, Genovese A, Trabilsy M, et al. Synthetic Patient–Physician Conversations Simulated by Large Language Models: A Multi-Dimensional Evaluation. Sensors. 2025. doi:10.3390/s25144305

BibTeX
@article{Haider2025,
  author = {Haider, Syed Ali and Prabha, Srinivasagam and Gomez-Cabello, Cesar Abraham and Borna, Sahar and Genovese, Ariana and Trabilsy, Maissa and Collaco, Bernardo G and Wood, Nadia G and Bagaria, Sanjay and Tao, Cui and Forte, Antonio Jorge},
  title = {Synthetic Patient–Physician Conversations Simulated by Large Language Models: A Multi-Dimensional Evaluation},
  journal = {Sensors},
  year = {2025},
  doi = {10.3390/s25144305}
}
Kostenlos starten →