EUVIMEDEUVIMED entdecken
Open Access

Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study

Yun Hao Leong iD, Lathiga Nambiar iD, Victoria Y J Tay iD, Sui An Lie iD, Ke Yuhe iD

JMIR Formative Research · 2025

DOI: 10.2196/77580 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Yun Hao Leong und 4 weiteren Autor:innen. Beteiligte Einrichtung: Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore. Hauptthemen laut MEDLINE: Educational Measurement, Education, Medical, Graduate, Anesthesiology, Language. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are increasingly used in medical education for feedback and grading; yet their role in postgraduate examination preparation remains uncertain due to inconsistent grading, hallucinations, and user acceptance.

Objective

This study evaluates the Personalized Anesthesia Study Support (PASS), a specialized GPT-4 model developed to assist candidates preparing for Singapore’s postgraduate specialist anesthesiology examination. We assessed user acceptance, grading interrater reliability, and hallucination detection rates to determine the feasibility of integrating specialized LLMs into high-stakes examination preparation.

Methods

PASS was built on OpenAI’s GPT-4 and adapted with domain-specific prompts and references. Twenty-one senior anesthesiology residents completed a mock short answer question examination, which was independently graded by 3 human examiners and 3 PASS iterations. Participants reviewed feedback from both PASS and standard GPT-4 and completed a technology acceptance model (TAM) survey. Grading reliability was evaluated using Cohen and Fleiss κ. Hallucination rates were assessed by participants and examiners.

Results

Of the 21 participants, 17 (81%) completed the TAM survey, generating 136 responses. PASS scored significantly higher than standard GPT-4 in usefulness (mean 4.25, SD 0.50 vs mean 3.44, SD 0.82; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Educational MeasurementEducation, Medical, GraduateAnesthesiologyLanguageHumansFeasibility StudiesSingaporeReproducibility of ResultsProof of Concept StudyFemaleInternship and ResidencyMaleAdultSurveys and QuestionnairesClinical CompetenceLarge Language Models

Stichwörter der Autor:innen

artificial intelligencelarge language modelmedical educationpostgraduate examinationtechnology acceptance model

Autor:innen und Institutionen

  • Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore.
  • Department of Anesthesiology, Sengkang General Hospital, Singapore, Singapore.

ORCID-Profile: Yun Hao Leong, Lathiga Nambiar, Victoria Y J Tay, Sui An Lie, Ke Yuhe

Zitieren

APA

Leong, Y. H., Nambiar, L., Tay, V. Y. J., Lie, S. A., & Yuhe, K. (2025). Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study. JMIR Formative Research. https://doi.org/10.2196/77580

Vancouver

Leong YH, Nambiar L, Tay VYJ, Lie SA, Yuhe K. Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study. JMIR Formative Research. 2025. doi:10.2196/77580

BibTeX
@article{Leong2025,
  author = {Leong, Yun Hao and Nambiar, Lathiga and Tay, Victoria Y J and Lie, Sui An and Yuhe, Ke},
  title = {Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/77580}
}
Kostenlos starten →