EUVIMEDEUVIMED entdecken
Open Access

Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment

Chenxu Wang iD, Shuhan Li iD, Nuoxi Lin iD, Xinyu Zhang iD, Ying Han iD, Xiandi Wang iD, Di Liu iD, Xiaomei Tan iD, Dan Pu iD, Kang Li iD, Guangwu Qian iD, Rong Yin iD

Journal of Medical Internet Research · 01.01.2025 · 27 · e59435

DOI: 10.2196/59435 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 01.01.2025. Verfasst von Chenxu Wang und 11 weiteren Autor:innen. Beteiligte Einrichtung: West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China. Hauptthemen laut MEDLINE: Education, Medical. Die Arbeit wurde bisher 65-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

With the increasing interest in the application of large language models (LLMs) in the medical field, the feasibility of its potential use as a standardized patient in medical assessment is rarely evaluated. Specifically, we delved into the potential of using ChatGPT, a representative LLM, in transforming medical education by serving as a cost-effective alternative to standardized patients, specifically for history-taking tasks.

Objective

The study aims to explore ChatGPT’s viability and performance as a standardized patient, using prompt engineering to refine its accuracy and use in medical assessments.

Methods

A 2-phase experiment was conducted. The first phase assessed feasibility by simulating conversations about inflammatory bowel disease (IBD) across 3 quality groups (good, medium, and bad). Responses were categorized based on their relevance and accuracy. Each group consisted of 30 runs, with responses scored to determine whether they were related to the inquiries. For the second phase, we evaluated ChatGPT’s performance against specific criteria, focusing on its anthropomorphism, clinical accuracy, and adaptability. Adjustments were made to prompts based on ChatGPT’s response shortcomings, with a comparative analysis of ChatGPT’s performance between original and revised prompts. A total of 300 runs were conducted and compared against standard reference scores. Finally, the generalizability of the revised prompt was tested using other scripts for another 60 runs, together with the exploration of the impact of the used language on the performance of the chatbot.

Results

The feasibility test confirmed ChatGPT’s ability to simulate a standardized patient effectively, differentiating among poor, medium, and good medical inquiries with varying degrees of accuracy. Score differences between the poor (74.7, SD 5.44) and medium (82.67, SD 5.30) inquiry groups (P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Education, MedicalHumansPatient SimulationInflammatory Bowel DiseasesLanguageFeasibility Studies

Stichwörter der Autor:innen

ChatGPTaccuracyartificial intelligencehealth careinflammatory bowel diseaselarge language modelsmedical trainingperformance evaluationprompt engineeringstandardized patient

Autor:innen und Institutionen

  • West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China.
  • Department of Industrial Engineering, Pittsburgh Institute, Sichuan University, Chengdu, China.
  • Department of Medical Simulation Center, West China Hospital, Sichuan University, Chengdu, China.
  • Med-X Center for Informatics, Sichuan University, Chengdu, China.
  • Department of Computer Science, Pittsburgh Institute, Sichuan University, Chengdu, China.

ORCID-Profile: Chenxu Wang, Shuhan Li, Nuoxi Lin, Xinyu Zhang, Ying Han, Xiandi Wang, Di Liu, Xiaomei Tan, Dan Pu, Kang Li, Guangwu Qian, Rong Yin

Zitieren

APA

Wang, C., Li, S., Lin, N., Zhang, X., Han, Y., Wang, X., Liu, D., Tan, X., Pu, D., Li, K., Qian, G., & Yin, R. (2025). Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment. Journal of Medical Internet Research, 27, e59435. https://doi.org/10.2196/59435

Vancouver

Wang C, Li S, Lin N, Zhang X, Han Y, Wang X, et al. Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment. Journal of Medical Internet Research. 2025;27:e59435. doi:10.2196/59435

BibTeX
@article{Wang2025,
  author = {Wang, Chenxu and Li, Shuhan and Lin, Nuoxi and Zhang, Xinyu and Han, Ying and Wang, Xiandi and Liu, Di and Tan, Xiaomei and Pu, Dan and Li, Kang and Qian, Guangwu and Yin, Rong},
  title = {Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e59435},
  doi = {10.2196/59435}
}
Kostenlos starten →