EUVIMEDEUVIMED entdecken
VergleichsstudieQuerschnittstudieOpen Access

Human-Edited Generative AI-Assisted Multiple-Choice Questions in Postgraduate Family Medicine: Blinded Cross-Sectional Comparative Psychometric Study

Kai Ping Sze iD, Jia Qing Lim iD, Yng Miin Loke iD, Teck Yong Gabriel Ding iD, Wei Chieh Wee iD, Tongyuan Chen iD, Jingkai Lawrence Wu iD, Vivek Bansal iD, Qi Wei Fong iD, Minyang Chow iD

JMIR Medical Education · 10.08.2026 · 12 · e100179-e100179

DOI: 10.2196/100179 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Medical Education, online erschienen am 10.08.2026. Verfasst von Kai Ping Sze und 9 weiteren Autor:innen. Beteiligte Einrichtung: LKC School of Medicine, Nanyang Technological University, Singapore, Singapore. Hauptthemen laut MEDLINE: Psychometrics, Educational Measurement, Family Practice. Die Arbeit nennt 22 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Generative artificial intelligence (GenAI) is increasingly used to draft multiple-choice questions (MCQs) for health professions education, but much evidence concerns raw model outputs, expert ratings, or item difficulty alone. Educators edit GenAI drafts before use, and whether such items are psychometrically ready for postgraduate assessment remains unclear.

Objective

This study aimed to compare human-edited GenAI-assisted and educator-crafted MCQs for postgraduate Family Medicine Applied Knowledge Test–level assessment, examining difficulty, discrimination, reliability, distractor functioning, and participant perceptions.

Methods

We conducted a blinded cross-sectional, within-participant comparative psychometric evaluation in Singapore. Sixty best-of-five single-best-answer MCQs were evaluated, 30 human-edited GenAI-assisted items and 30 educator-crafted items, topic-matched across postgraduate FM domains and randomized across 2 assessment sets. Eligible participants were postgraduate doctors enrolled in FM residency or postgraduate family medicine programs, preparing for the Applied Knowledge Test, and blinded to item origin; incomplete paired responses were excluded. Outcomes included paired total scores, score correlation and agreement, Kuder-Richardson Formula 20 reliability, item difficulty index, corrected point-biserial discrimination, distractor functioning, and perceived difficulty, clarity, and relevance. Analyses used paired-sample tests, Pearson correlation, Fisher exact tests, and item-level psychometric statistics, with α=.05 and Bonferroni correction within comparison families.

Results

Of 74 participants, 73 completed both item sets and were included in the analysis. The final sample comprised 36 graduate diploma in FM trainees, 5 MMed FM trainees, and 32 FM residents. Paired-sample testing showed lower scores on GenAI-assisted than educator-crafted items (mean 19.12, SD 2.83 vs mean 21.10, SD 3.42 out of 30; mean difference −1.97, 95% CI −2.72 to −1.23; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

PsychometricsEducational MeasurementFamily PracticeHumansCross-Sectional StudiesGenerative Artificial IntelligenceMaleFemaleReproducibility of ResultsSingaporeAdultEducation, Medical, Graduate

Stichwörter der Autor:innen

ChatGPTassessmentdistractor analysisfamily medicinegenerative artificial intelligenceitem analysismedical educationmultiple-choice questionspsychometrics

Autor:innen und Institutionen

  • LKC School of Medicine, Nanyang Technological University, Singapore, Singapore.
  • NHG Polyclinics, Population Health Campus, National Healthcare Group, 690 Yishun Ring Road, Singapore, 769567, Singapore, 65 63553000.
  • Clinical Education, National Healthcare Group, Singapore, Singapore.
  • Professional Training and Standards Division, Ministry of Health, Singapore, Singapore, Singapore.

ORCID-Profile: Kai Ping Sze, Jia Qing Lim, Yng Miin Loke, Teck Yong Gabriel Ding, Wei Chieh Wee, Tongyuan Chen, Jingkai Lawrence Wu, Vivek Bansal, Qi Wei Fong, Minyang Chow

Zitieren

APA

Sze, K. P., Lim, J. Q., Loke, Y. M., Ding, T. Y. G., Wee, W. C., Chen, T., Wu, J. L., Bansal, V., Fong, Q. W., & Chow, M. (2026). Human-Edited Generative AI-Assisted Multiple-Choice Questions in Postgraduate Family Medicine: Blinded Cross-Sectional Comparative Psychometric Study. JMIR Medical Education, 12, e100179-e100179. https://doi.org/10.2196/100179

Vancouver

Sze KP, Lim JQ, Loke YM, Ding TYG, Wee WC, Chen T, et al. Human-Edited Generative AI-Assisted Multiple-Choice Questions in Postgraduate Family Medicine: Blinded Cross-Sectional Comparative Psychometric Study. JMIR Medical Education. 2026;12:e100179-e100179. doi:10.2196/100179

BibTeX
@article{Sze2026,
  author = {Sze, Kai Ping and Lim, Jia Qing and Loke, Yng Miin and Ding, Teck Yong Gabriel and Wee, Wei Chieh and Chen, Tongyuan and Wu, Jingkai Lawrence and Bansal, Vivek and Fong, Qi Wei and Chow, Minyang},
  title = {Human-Edited Generative AI-Assisted Multiple-Choice Questions in Postgraduate Family Medicine: Blinded Cross-Sectional Comparative Psychometric Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e100179--e100179},
  doi = {10.2196/100179}
}
Kostenlos starten →