Evaluating Google Gemini’s Capability to Generate NBME-Standard Pharmacology Questions Using a 16-Criterion NBME Rubric
Algorithms · 29.09.2025 · 18 (10) · 612
DOI: 10.3390/a18100612 ↗Kurz gefasst
Fachartikel in der Zeitschrift Algorithms, online erschienen am 29.09.2025. Verfasst von Wesam Almasri und 2 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Large language models (LLMs) such as Google Gemini have demonstrated strong capabilities in natural language generation, but their ability to create medical assessment items aligned with National Board of Medical Examiners (NBME) standards remains underexplored.
Objective
This study evaluated the quality of Gemini-generated NBME-style pharmacology questions using a structured rubric to assess accuracy, clarity, and alignment with examination standards.
Methods
Ten pharmacology questions were generated using a standardized prompt and assessed independently by two pharmacology experts. Each item was evaluated using a 16-criterion NBME rubric with binary scoring. Inter-rater reliability was calculated (Cohen’s Kappa = 0.81) following a calibration session.
Results
On average, questions met 14.3 of 16 criteria. Strengths included logical structure, appropriate distractors, and clinically relevant framing. Limitations included occasional pseudo-vignettes, cueing issues, and one instance of factual inaccuracy (albuterol mechanism of action). The evaluation highlighted Gemini’s ability to produce high-quality NBME-style questions, while underscoring concerns regarding sample size, reproducibility, and factual reliability.
Conclusions
Gemini shows promise as a tool for generating pharmacology assessment items, but its probabilistic outputs, factual inaccuracies, and limited scope necessitate caution. Larger-scale studies, inclusion of multiple medical disciplines, incorporation of student performance data, and use of broader expert panels are recommended to establish reliability and educational applicability.
Zitieren
APA
Almasri, W., Saad, M., & Mohiyeddini, C. (2025). Evaluating Google Gemini’s Capability to Generate NBME-Standard Pharmacology Questions Using a 16-Criterion NBME Rubric. Algorithms, 18(10), 612. https://doi.org/10.3390/a18100612
Vancouver
Almasri W, Saad M, Mohiyeddini C. Evaluating Google Gemini’s Capability to Generate NBME-Standard Pharmacology Questions Using a 16-Criterion NBME Rubric. Algorithms. 2025;18(10):612. doi:10.3390/a18100612
BibTeX
@article{Almasri2025,
author = {Almasri, Wesam and Saad, Marwa and Mohiyeddini, Changiz},
title = {Evaluating Google Gemini’s Capability to Generate NBME-Standard Pharmacology Questions Using a 16-Criterion NBME Rubric},
journal = {Algorithms},
year = {2025},
volume = {18},
number = {10},
pages = {612},
doi = {10.3390/a18100612}
}
EUVIMED