EUVIMEDScopri EUVIMED
Open access

Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study

Yun Hao Leong iD, Lathiga Nambiar iD, Victoria Y J Tay iD, Sui An Lie iD, Ke Yuhe iD

JMIR Formative Research · 2025

DOI: 10.2196/77580 ↗

At a glance

Journal article in JMIR Formative Research, 2025. Written by Yun Hao Leong and 4 co-authors. Institution: Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore. Major MEDLINE topics: Educational Measurement, Education, Medical, Graduate, Anesthesiology, Language. Cited 5 times so far (Crossref); 32 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Large language models (LLMs) are increasingly used in medical education for feedback and grading; yet their role in postgraduate examination preparation remains uncertain due to inconsistent grading, hallucinations, and user acceptance.

Objective

This study evaluates the Personalized Anesthesia Study Support (PASS), a specialized GPT-4 model developed to assist candidates preparing for Singapore’s postgraduate specialist anesthesiology examination. We assessed user acceptance, grading interrater reliability, and hallucination detection rates to determine the feasibility of integrating specialized LLMs into high-stakes examination preparation.

Methods

PASS was built on OpenAI’s GPT-4 and adapted with domain-specific prompts and references. Twenty-one senior anesthesiology residents completed a mock short answer question examination, which was independently graded by 3 human examiners and 3 PASS iterations. Participants reviewed feedback from both PASS and standard GPT-4 and completed a technology acceptance model (TAM) survey. Grading reliability was evaluated using Cohen and Fleiss κ. Hallucination rates were assessed by participants and examiners.

Results

Of the 21 participants, 17 (81%) completed the TAM survey, generating 136 responses. PASS scored significantly higher than standard GPT-4 in usefulness (mean 4.25, SD 0.50 vs mean 3.44, SD 0.82; P

Subjects

MeSH terms (MEDLINE)

Educational MeasurementEducation, Medical, GraduateAnesthesiologyLanguageHumansFeasibility StudiesSingaporeReproducibility of ResultsProof of Concept StudyFemaleInternship and ResidencyMaleAdultSurveys and QuestionnairesClinical CompetenceLarge Language Models

Author keywords

artificial intelligencelarge language modelmedical educationpostgraduate examinationtechnology acceptance model

Authors and institutions

  • Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore.
  • Department of Anesthesiology, Sengkang General Hospital, Singapore, Singapore.

ORCID profiles: Yun Hao Leong, Lathiga Nambiar, Victoria Y J Tay, Sui An Lie, Ke Yuhe

Cite

APA

Leong, Y. H., Nambiar, L., Tay, V. Y. J., Lie, S. A., & Yuhe, K. (2025). Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study. JMIR Formative Research. https://doi.org/10.2196/77580

Vancouver

Leong YH, Nambiar L, Tay VYJ, Lie SA, Yuhe K. Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study. JMIR Formative Research. 2025. doi:10.2196/77580

BibTeX
@article{Leong2025,
  author = {Leong, Yun Hao and Nambiar, Lathiga and Tay, Victoria Y J and Lie, Sui An and Yuhe, Ke},
  title = {Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/77580}
}
Start for free →