EUVIMEDEUVIMED entdecken
Open Access

Fine-Tuned Large Language Models for Generating Multiple-Choice Questions in Anesthesiology: Psychometric Comparison With Faculty-Written Items

Carlos Ramon Hölzing iD, Charlotte Meynhardt iD, Patrick Meybohm iD, Sarah König iD, Peter Kranke iD

JMIR Formative Research · 2026

DOI: 10.2196/84904 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, 2026. Verfasst von Carlos Ramon Hölzing und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Anaesthesiology, Intensive Care, Emergency and Pain Medicine, University Hospital Würzburg, Oberdürrbacher Str. 6, Würzburg, 97080, Ger…. Hauptthemen laut MEDLINE: Psychometrics, Educational Measurement, Anesthesiology. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Multiple-choice examinations (MCQs) are widely used in medical education to ensure standardized and objective assessment. Developing high-quality items requires both subject expertise and methodological rigor. Large language models (LLMs) offer new opportunities for automated item generation. However, most evaluations rely on general-purpose prompting, and psychometric comparisons with faculty-written items remain scarce.

Objective

This study aimed to evaluate whether a fine-tuned LLM can generate MCQs (Type A) in anesthesiology with psychometric properties comparable to those written by expert faculty.

Methods

The study was embedded in the regular written anesthesiology examination of the eighth-semester medical curriculum with 157 students. The examination comprised 30 single best-answer MCQs, of which 15 were generated by senior faculty and 15 by a fine-tuned GPT-based model. A custom GPT-based (GPT-4) model was adapted with anesthesiology lecture slides, the National Competence-Based Learning Objectives Catalogue (NKLM 2.0), past examination questions, and faculty publications using supervised instruction-tuning with standardized prompt–response pairs. Item analysis followed established psychometric standards.

Results

In total, 29 items (14 expert, 15 LLM-generated) were analyzed. Expert-generated questions had a mean difficulty of 0.81 (SD 0.19), point-biserial correlation of 0.19 (SD 0.07), and discrimination index of 0.09 (SD 0.08). LLM-generated items had a mean difficulty of 0.79 (SD 0.18), point-biserial correlation of 0.17 (SD 0.04), and discrimination index of 0.08 (SD 0.11). Mann-Whitney U tests revealed no significant differences between expert- and LLM-generated items for difficulty ( P =.38), point-biserial correlation coefficient ( P =.96), or discrimination index ( P =.59). Categorical analyses confirmed no significant group differences. Both sets, however, showed only modest psychometric quality.

Conclusions

Supervised fine-tuned LLMs are capable of generating MCQs with psychometric properties comparable to those written by experienced faculty. Given the limitations and cohort-dependency of psychometric indices, automated item generation should be considered a complement rather than a replacement for manual item writing. Further research with larger item sets and multi-institutional validation is needed to confirm generalizability and optimize integration of LLM-based tools into assessment development.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

PsychometricsEducational MeasurementAnesthesiologyLarge Language ModelsHumansFaculty, Medical

Stichwörter der Autor:innen

anesthesiologyartificial intelligenceassessmentfine-tuninglarge language modelsmedical educationmultiple-choice questionspsychometrics

Autor:innen und Institutionen

  • Department of Anaesthesiology, Intensive Care, Emergency and Pain Medicine, University Hospital Würzburg, Oberdürrbacher Str. 6, Würzburg, 97080, Germany.
  • Institute of Medical Teaching and Medical Education Research, University Hospital Würzburg, Würzburg, Germany.

ORCID-Profile: Carlos Ramon Hölzing, Charlotte Meynhardt, Patrick Meybohm, Sarah König, Peter Kranke

Zitieren

APA

Hölzing, C. R., Meynhardt, C., Meybohm, P., König, S., & Kranke, P. (2026). Fine-Tuned Large Language Models for Generating Multiple-Choice Questions in Anesthesiology: Psychometric Comparison With Faculty-Written Items. JMIR Formative Research. https://doi.org/10.2196/84904

Vancouver

Hölzing CR, Meynhardt C, Meybohm P, König S, Kranke P. Fine-Tuned Large Language Models for Generating Multiple-Choice Questions in Anesthesiology: Psychometric Comparison With Faculty-Written Items. JMIR Formative Research. 2026. doi:10.2196/84904

BibTeX
@article{Hlzing2026,
  author = {Hölzing, Carlos Ramon and Meynhardt, Charlotte and Meybohm, Patrick and König, Sarah and Kranke, Peter},
  title = {Fine-Tuned Large Language Models for Generating Multiple-Choice Questions in Anesthesiology: Psychometric Comparison With Faculty-Written Items},
  journal = {JMIR Formative Research},
  year = {2026},
  doi = {10.2196/84904}
}
Kostenlos starten →