Evaluating the instrumental quality of LLM-generated assessment items
Frontiers in Education · 23.06.2026 · 11
DOI: 10.3389/feduc.2026.1837523 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Education, online erschienen am 23.06.2026. Verfasst von Igor Crk und 1 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objective
This study compared the use of Large Language Models (LLMs) to produce Multiple-Choice Questions (MCQs) against human written MCQs for evaluating student performance on course material.
Methods
Assessment data was collected from an 8-week, synchronous online, third-year Operating Systems course from two successive cohorts (2024 and 2025) across a total of 42 students. 21 human and 14 LLM MCQs were created. The LLM-authored MCQs (with ChatGPT 4.0) used an expert-validation zero-shot protocol designed to match a human question on a topic. Various psychometric measures were computed and compared for both human and LLM-authored MCQs. Student performance was assessed using Item Difficulty. Discriminatory ability of the MCQs was assessed using the Discrimination Index and Corrected Point-Biserial Correlation. MCQ reliability was assessed using Krippendorf’s α , Item Reliability Index, correlations between student MCQ performance to course and project performance, and network psychometric techniques. Network analyses were performed using Exploratory Graph Analysis and Network Comparison Tests.
Results
The Item Difficulty scores were high ( > 0.80 ) for LLM and human MCQs indicating that they were relatively straightforward, but not statistical different. The Discrimination Index, Corrected Point-Biserial Correlation, and Item Reliability Index values were low, but not statistically different. Krippendorf’s α across all Human ( α = 0.695 ) and LLM-authored ( α = 0.734 ) questions were moderate and not statistically different. Network comparison suggested that LLM-authored MCQs replicated latent knowledge structures in the human MCQs by exhibiting no statistically significant difference.
Conclusion
The study provides evidence that zero-shot expert-validated LLM output can produce MCQs that are fair, well-composed, and relevant, especially for computer science related subjects.
Zitieren
APA
Crk, I., & Gultepe, E. (2026). Evaluating the instrumental quality of LLM-generated assessment items. Frontiers in Education, 11. https://doi.org/10.3389/feduc.2026.1837523
Vancouver
Crk I, Gultepe E. Evaluating the instrumental quality of LLM-generated assessment items. Frontiers in Education. 2026;11. doi:10.3389/feduc.2026.1837523
BibTeX
@article{Crk2026,
author = {Crk, Igor and Gultepe, Eren},
title = {Evaluating the instrumental quality of LLM-generated assessment items},
journal = {Frontiers in Education},
year = {2026},
volume = {11},
doi = {10.3389/feduc.2026.1837523}
}
EUVIMED