EUVIMEDDescoperă EUVIMED
Open access

Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory Study

Seysha Mehta iD, Eliot N Haddad iD, Indira Bhavsar Burke iD, Alana K Majors iD, Rie Maeda iD, Sean M Burke iD, Abhishek Deshpande iD, Amy S Nowacki iD, Christina C Lindenmeyer iD, Neil Mehta iD

JMIR Medical Education · 2025-06-16 · 11 · e72034-e72034

DOI: 10.2196/72034 ↗

At a glance

Journal article in JMIR Medical Education, published online on 2025-06-16. Written by Seysha Mehta and 9 co-authors. Institution: Cleveland Clinic Lerner College of Medicine, School of Medicine, Case Western Reserve University, 9500 Euclid Ave, G10, Cleveland, OH, 44195, United …. Major MEDLINE topics: Students, Medical, Language, Educational Measurement, Education, Medical, Undergraduate. Cited 4 times so far (Crossref); 10 references. The full text is freely available in PubMed Central.

RezumatSubjectsDate bibliograficeCiteRelated studies

Abstract (original language)

Bing Chat (subsequently renamed Microsoft Copilot)—a ChatGPT 4.0–based large language model—demonstrated comparable performance to medical students in answering essay-style concept appraisals, while assessors struggled to differentiate artificial intelligence (AI) responses from human responses. These results highlight the need to prepare students and educators for a future world of AI by fostering reflective learning practices and critical thinking.

Subjects

MeSH terms (MEDLINE)

Students, MedicalLanguageEducational MeasurementEducation, Medical, UndergraduateHumansArtificial IntelligenceSchools, MedicalLarge Language Models

Author keywords

Microsoft Copilotartificial intelligenceessay-type questionsgenerative AIlarge language models

Authors and institutions

  • Cleveland Clinic Lerner College of Medicine, School of Medicine, Case Western Reserve University, 9500 Euclid Ave, G10, Cleveland, OH, 44195, United States, 1 2164456512, 1 2164451007.
  • Department of Internal Medicine, The University of Texas Southwestern Medical Center, Dallas, TX, United States.

ORCID profiles: Seysha Mehta, Eliot N Haddad, Indira Bhavsar Burke, Alana K Majors, Rie Maeda, Sean M Burke, Abhishek Deshpande, Amy S Nowacki, Christina C Lindenmeyer, Neil Mehta

Cite

APA

Mehta, S., Haddad, E. N., Burke, I. B., Majors, A. K., Maeda, R., Burke, S. M., Deshpande, A., Nowacki, A. S., Lindenmeyer, C. C., & Mehta, N. (2025). Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory Study. JMIR Medical Education, 11, e72034-e72034. https://doi.org/10.2196/72034

Vancouver

Mehta S, Haddad EN, Burke IB, Majors AK, Maeda R, Burke SM, et al. Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory Study. JMIR Medical Education. 2025;11:e72034-e72034. doi:10.2196/72034

BibTeX
@article{Mehta2025,
  author = {Mehta, Seysha and Haddad, Eliot N and Burke, Indira Bhavsar and Majors, Alana K and Maeda, Rie and Burke, Sean M and Deshpande, Abhishek and Nowacki, Amy S and Lindenmeyer, Christina C and Mehta, Neil},
  title = {Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory Study},
  journal = {JMIR Medical Education},
  year = {2025},
  volume = {11},
  pages = {e72034--e72034},
  doi = {10.2196/72034}
}
Start for free →