Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study
JMIR Formative Research · 2025
DOI: 10.2196/77580 ↗At a glance
Journal article in JMIR Formative Research, 2025. Written by Yun Hao Leong and 4 co-authors. Institution: Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore. Major MEDLINE topics: Educational Measurement, Education, Medical, Graduate, Anesthesiology, Language. Cited 5 times so far (Crossref); 32 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Large language models (LLMs) are increasingly used in medical education for feedback and grading; yet their role in postgraduate examination preparation remains uncertain due to inconsistent grading, hallucinations, and user acceptance.
Objective
This study evaluates the Personalized Anesthesia Study Support (PASS), a specialized GPT-4 model developed to assist candidates preparing for Singapore’s postgraduate specialist anesthesiology examination. We assessed user acceptance, grading interrater reliability, and hallucination detection rates to determine the feasibility of integrating specialized LLMs into high-stakes examination preparation.
Methods
PASS was built on OpenAI’s GPT-4 and adapted with domain-specific prompts and references. Twenty-one senior anesthesiology residents completed a mock short answer question examination, which was independently graded by 3 human examiners and 3 PASS iterations. Participants reviewed feedback from both PASS and standard GPT-4 and completed a technology acceptance model (TAM) survey. Grading reliability was evaluated using Cohen and Fleiss κ. Hallucination rates were assessed by participants and examiners.
Results
Of the 21 participants, 17 (81%) completed the TAM survey, generating 136 responses. PASS scored significantly higher than standard GPT-4 in usefulness (mean 4.25, SD 0.50 vs mean 3.44, SD 0.82; P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore.
- Department of Anesthesiology, Sengkang General Hospital, Singapore, Singapore.
ORCID profiles: Yun Hao Leong, Lathiga Nambiar, Victoria Y J Tay, Sui An Lie, Ke Yuhe
Cite
APA
Leong, Y. H., Nambiar, L., Tay, V. Y. J., Lie, S. A., & Yuhe, K. (2025). Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study. JMIR Formative Research. https://doi.org/10.2196/77580
Vancouver
Leong YH, Nambiar L, Tay VYJ, Lie SA, Yuhe K. Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study. JMIR Formative Research. 2025. doi:10.2196/77580
BibTeX
@article{Leong2025,
author = {Leong, Yun Hao and Nambiar, Lathiga and Tay, Victoria Y J and Lie, Sui An and Yuhe, Ke},
title = {Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study},
journal = {JMIR Formative Research},
year = {2025},
doi = {10.2196/77580}
}
EUVIMED