Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine
SN Computer Science · 10.06.2024 · 5 (5)
DOI: 10.1007/s42979-024-02963-6 ↗Kurz gefasst
Fachartikel in der Zeitschrift SN Computer Science, online erschienen am 10.06.2024. Verfasst von Christian Grévisse und 2 weiteren Autor:innen. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Assessment is an essential part of education, both for teachers who assess their students as well as learners who may evaluate themselves. Multiple-choice questions (MCQ) are one of the most popular types of knowledge assessment, e.g., in medical education, as they can be automatically graded and can cover a wide range of learning items. However, the creation of high-quality MCQ items is a time-consuming task. The recent advent of Large Language Models (LLM), such as Generative Pre-trained Transformer (GPT), caused a new momentum for automatic question generation solutions. Still, evaluating generated questions according to the best practices for MCQ item writing is needed to ensure docimological quality. In this article, we propose an analysis of the quality of LLM-generated MCQs. We employ zero-shot approaches in two domains, namely computer science and medicine. In the former, we make use of 3 GPT-based services to generate MCQs. In the latter, we developed a plugin for the Moodle learning management system that generates MCQs based on learning material. We compare the generated MCQs against common multiple-choice item writing guidelines. Among the major challenges, we determined that while LLMs are certainly useful in generating MCQs more efficiently, they sometimes create broad items with ambiguous keys or implausible distractors. Human oversight is also necessary to ensure instructional alignment between generated items and course contents. Finally, we propose solutions for AQG developers.
Zitieren
APA
Grévisse, C., Pavlou, M. A. S., & Schneider, J. G. (2024). Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine. SN Computer Science, 5(5). https://doi.org/10.1007/s42979-024-02963-6
Vancouver
Grévisse C, Pavlou MAS, Schneider JG. Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine. SN Computer Science. 2024;5(5). doi:10.1007/s42979-024-02963-6
BibTeX
@article{Grvisse2024,
author = {Grévisse, Christian and Pavlou, Maria Angeliki S. and Schneider, Jochen G.},
title = {Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine},
journal = {SN Computer Science},
year = {2024},
volume = {5},
number = {5},
doi = {10.1007/s42979-024-02963-6}
}
EUVIMED