Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment
Journal of Medical Internet Research · 01.01.2025 · 27 · e59435
DOI: 10.2196/59435 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 01.01.2025. Verfasst von Chenxu Wang und 11 weiteren Autor:innen. Beteiligte Einrichtung: West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China. Hauptthemen laut MEDLINE: Education, Medical. Die Arbeit wurde bisher 65-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
With the increasing interest in the application of large language models (LLMs) in the medical field, the feasibility of its potential use as a standardized patient in medical assessment is rarely evaluated. Specifically, we delved into the potential of using ChatGPT, a representative LLM, in transforming medical education by serving as a cost-effective alternative to standardized patients, specifically for history-taking tasks.
Objective
The study aims to explore ChatGPT’s viability and performance as a standardized patient, using prompt engineering to refine its accuracy and use in medical assessments.
Methods
A 2-phase experiment was conducted. The first phase assessed feasibility by simulating conversations about inflammatory bowel disease (IBD) across 3 quality groups (good, medium, and bad). Responses were categorized based on their relevance and accuracy. Each group consisted of 30 runs, with responses scored to determine whether they were related to the inquiries. For the second phase, we evaluated ChatGPT’s performance against specific criteria, focusing on its anthropomorphism, clinical accuracy, and adaptability. Adjustments were made to prompts based on ChatGPT’s response shortcomings, with a comparative analysis of ChatGPT’s performance between original and revised prompts. A total of 300 runs were conducted and compared against standard reference scores. Finally, the generalizability of the revised prompt was tested using other scripts for another 60 runs, together with the exploration of the impact of the used language on the performance of the chatbot.
Results
The feasibility test confirmed ChatGPT’s ability to simulate a standardized patient effectively, differentiating among poor, medium, and good medical inquiries with varying degrees of accuracy. Score differences between the poor (74.7, SD 5.44) and medium (82.67, SD 5.30) inquiry groups (P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China.
- Department of Industrial Engineering, Pittsburgh Institute, Sichuan University, Chengdu, China.
- Department of Medical Simulation Center, West China Hospital, Sichuan University, Chengdu, China.
- Med-X Center for Informatics, Sichuan University, Chengdu, China.
- Department of Computer Science, Pittsburgh Institute, Sichuan University, Chengdu, China.
ORCID-Profile: Chenxu Wang, Shuhan Li, Nuoxi Lin, Xinyu Zhang, Ying Han, Xiandi Wang, Di Liu, Xiaomei Tan, Dan Pu, Kang Li, Guangwu Qian, Rong Yin
Zitieren
APA
Wang, C., Li, S., Lin, N., Zhang, X., Han, Y., Wang, X., Liu, D., Tan, X., Pu, D., Li, K., Qian, G., & Yin, R. (2025). Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment. Journal of Medical Internet Research, 27, e59435. https://doi.org/10.2196/59435
Vancouver
Wang C, Li S, Lin N, Zhang X, Han Y, Wang X, et al. Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment. Journal of Medical Internet Research. 2025;27:e59435. doi:10.2196/59435
BibTeX
@article{Wang2025,
author = {Wang, Chenxu and Li, Shuhan and Lin, Nuoxi and Zhang, Xinyu and Han, Ying and Wang, Xiandi and Liu, Di and Tan, Xiaomei and Pu, Dan and Li, Kang and Qian, Guangwu and Yin, Rong},
title = {Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e59435},
doi = {10.2196/59435}
}
EUVIMED