Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory Study
JMIR Medical Education · 2025-06-16 · 11 · e72034-e72034
DOI: 10.2196/72034 ↗At a glance
Journal article in JMIR Medical Education, published online on 2025-06-16. Written by Seysha Mehta and 9 co-authors. Institution: Cleveland Clinic Lerner College of Medicine, School of Medicine, Case Western Reserve University, 9500 Euclid Ave, G10, Cleveland, OH, 44195, United …. Major MEDLINE topics: Students, Medical, Language, Educational Measurement, Education, Medical, Undergraduate. Cited 4 times so far (Crossref); 10 references. The full text is freely available in PubMed Central.
Abstract (original language)
Bing Chat (subsequently renamed Microsoft Copilot)—a ChatGPT 4.0–based large language model—demonstrated comparable performance to medical students in answering essay-style concept appraisals, while assessors struggled to differentiate artificial intelligence (AI) responses from human responses. These results highlight the need to prepare students and educators for a future world of AI by fostering reflective learning practices and critical thinking.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Cleveland Clinic Lerner College of Medicine, School of Medicine, Case Western Reserve University, 9500 Euclid Ave, G10, Cleveland, OH, 44195, United States, 1 2164456512, 1 2164451007.
- Department of Internal Medicine, The University of Texas Southwestern Medical Center, Dallas, TX, United States.
ORCID profiles: Seysha Mehta, Eliot N Haddad, Indira Bhavsar Burke, Alana K Majors, Rie Maeda, Sean M Burke, Abhishek Deshpande, Amy S Nowacki, Christina C Lindenmeyer, Neil Mehta
Cite
APA
Mehta, S., Haddad, E. N., Burke, I. B., Majors, A. K., Maeda, R., Burke, S. M., Deshpande, A., Nowacki, A. S., Lindenmeyer, C. C., & Mehta, N. (2025). Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory Study. JMIR Medical Education, 11, e72034-e72034. https://doi.org/10.2196/72034
Vancouver
Mehta S, Haddad EN, Burke IB, Majors AK, Maeda R, Burke SM, et al. Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory Study. JMIR Medical Education. 2025;11:e72034-e72034. doi:10.2196/72034
BibTeX
@article{Mehta2025,
author = {Mehta, Seysha and Haddad, Eliot N and Burke, Indira Bhavsar and Majors, Alana K and Maeda, Rie and Burke, Sean M and Deshpande, Abhishek and Nowacki, Amy S and Lindenmeyer, Christina C and Mehta, Neil},
title = {Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory Study},
journal = {JMIR Medical Education},
year = {2025},
volume = {11},
pages = {e72034--e72034},
doi = {10.2196/72034}
}
EUVIMED