EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating the instrumental quality of LLM-generated assessment items

Igor Crk, Eren Gultepe

Frontiers in Education · 23.06.2026 · 11

DOI: 10.3389/feduc.2026.1837523 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Education, online erschienen am 23.06.2026. Verfasst von Igor Crk und 1 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

This study compared the use of Large Language Models (LLMs) to produce Multiple-Choice Questions (MCQs) against human written MCQs for evaluating student performance on course material.

Methods

Assessment data was collected from an 8-week, synchronous online, third-year Operating Systems course from two successive cohorts (2024 and 2025) across a total of 42 students. 21 human and 14 LLM MCQs were created. The LLM-authored MCQs (with ChatGPT 4.0) used an expert-validation zero-shot protocol designed to match a human question on a topic. Various psychometric measures were computed and compared for both human and LLM-authored MCQs. Student performance was assessed using Item Difficulty. Discriminatory ability of the MCQs was assessed using the Discrimination Index and Corrected Point-Biserial Correlation. MCQ reliability was assessed using Krippendorf’s α , Item Reliability Index, correlations between student MCQ performance to course and project performance, and network psychometric techniques. Network analyses were performed using Exploratory Graph Analysis and Network Comparison Tests.

Results

The Item Difficulty scores were high ( > 0.80 ) for LLM and human MCQs indicating that they were relatively straightforward, but not statistical different. The Discrimination Index, Corrected Point-Biserial Correlation, and Item Reliability Index values were low, but not statistically different. Krippendorf’s α across all Human ( α = 0.695 ) and LLM-authored ( α = 0.734 ) questions were moderate and not statistically different. Network comparison suggested that LLM-authored MCQs replicated latent knowledge structures in the human MCQs by exhibiting no statistically significant difference.

Conclusion

The study provides evidence that zero-shot expert-validated LLM output can produce MCQs that are fair, well-composed, and relevant, especially for computer science related subjects.

Zitieren

APA

Crk, I., & Gultepe, E. (2026). Evaluating the instrumental quality of LLM-generated assessment items. Frontiers in Education, 11. https://doi.org/10.3389/feduc.2026.1837523

Vancouver

Crk I, Gultepe E. Evaluating the instrumental quality of LLM-generated assessment items. Frontiers in Education. 2026;11. doi:10.3389/feduc.2026.1837523

BibTeX
@article{Crk2026,
  author = {Crk, Igor and Gultepe, Eren},
  title = {Evaluating the instrumental quality of LLM-generated assessment items},
  journal = {Frontiers in Education},
  year = {2026},
  volume = {11},
  doi = {10.3389/feduc.2026.1837523}
}
Kostenlos starten →