Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation Using Knowledge Graphs and Large Language Models
Machine Learning and Knowledge Extraction · 2026-05-20 · 8 (5) · 137
DOI: 10.3390/make8050137 ↗At a glance
Journal article in Machine Learning and Knowledge Extraction, published online on 2026-05-20. Written by Mehmet Can Şakiroğlu and 2 co-authors. 33 references. The full text is freely available (open access, CC BY 4.0). Funded by: European Union.
Abstract (original language)
Generating multiple-choice questions (MCQs) with difficulty estimation remains challenging in automated MCQ-generation systems used in adaptive, AI-assisted education. This study proposes a novel methodology for generating MCQs with difficulty estimation from input documents by utilizing knowledge graphs (KGs) and large language models (LLMs). Our approach uses an LLM to construct a KG from input documents, from which MCQs are then systematically generated. Each MCQ is generated by selecting a node from the KG as the key, sampling a related triple or quintuple—optionally augmented with an extra triple—and prompting an LLM to generate a corresponding stem from these graph components. Distractors are then selected from the KG. For each MCQ, nine difficulty signals are computed and combined into a unified difficulty score using a data-driven approach. Within a 150-MCQ, proof-of-concept dataset from Wikipedia, the proposed signals show interpretable associations with empirical incorrect-answer rates aligning with human responses/performance. The results support the feasibility of the proposed pipeline, yet a larger-scale human study may be required to establish deployment-scale validity. Our approach improves automated MCQ generation by integrating structured knowledge representations with LLMs and a data-driven difficulty estimation model.
Funding
- European Union
Cite
APA
Şakiroğlu, M. C., Güvenir, H. A., & Kaya, K. (2026). Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation Using Knowledge Graphs and Large Language Models. Machine Learning and Knowledge Extraction, 8(5), 137. https://doi.org/10.3390/make8050137
Vancouver
Şakiroğlu MC, Güvenir HA, Kaya K. Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation Using Knowledge Graphs and Large Language Models. Machine Learning and Knowledge Extraction. 2026;8(5):137. doi:10.3390/make8050137
BibTeX
@article{akirolu2026,
author = {Şakiroğlu, Mehmet Can and Güvenir, Halil Altay and Kaya, Kamer},
title = {Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation Using Knowledge Graphs and Large Language Models},
journal = {Machine Learning and Knowledge Extraction},
year = {2026},
volume = {8},
number = {5},
pages = {137},
doi = {10.3390/make8050137}
}
EUVIMED