EUVIMEDOntdek EUVIMED
Open access · CC BY 4.0

Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation Using Knowledge Graphs and Large Language Models

Mehmet Can Şakiroğlu, Halil Altay Güvenir, Kamer Kaya

Machine Learning and Knowledge Extraction · 2026-05-20 · 8 (5) · 137

DOI: 10.3390/make8050137 ↗

At a glance

Journal article in Machine Learning and Knowledge Extraction, published online on 2026-05-20. Written by Mehmet Can Şakiroğlu and 2 co-authors. 33 references. The full text is freely available (open access, CC BY 4.0). Funded by: European Union.

SamenvattingBibliografische gegevensCiteRelated studies

Abstract (original language)

Generating multiple-choice questions (MCQs) with difficulty estimation remains challenging in automated MCQ-generation systems used in adaptive, AI-assisted education. This study proposes a novel methodology for generating MCQs with difficulty estimation from input documents by utilizing knowledge graphs (KGs) and large language models (LLMs). Our approach uses an LLM to construct a KG from input documents, from which MCQs are then systematically generated. Each MCQ is generated by selecting a node from the KG as the key, sampling a related triple or quintuple—optionally augmented with an extra triple—and prompting an LLM to generate a corresponding stem from these graph components. Distractors are then selected from the KG. For each MCQ, nine difficulty signals are computed and combined into a unified difficulty score using a data-driven approach. Within a 150-MCQ, proof-of-concept dataset from Wikipedia, the proposed signals show interpretable associations with empirical incorrect-answer rates aligning with human responses/performance. The results support the feasibility of the proposed pipeline, yet a larger-scale human study may be required to establish deployment-scale validity. Our approach improves automated MCQ generation by integrating structured knowledge representations with LLMs and a data-driven difficulty estimation model.

Funding

  • European Union

Cite

APA

Şakiroğlu, M. C., Güvenir, H. A., & Kaya, K. (2026). Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation Using Knowledge Graphs and Large Language Models. Machine Learning and Knowledge Extraction, 8(5), 137. https://doi.org/10.3390/make8050137

Vancouver

Şakiroğlu MC, Güvenir HA, Kaya K. Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation Using Knowledge Graphs and Large Language Models. Machine Learning and Knowledge Extraction. 2026;8(5):137. doi:10.3390/make8050137

BibTeX
@article{akirolu2026,
  author = {Şakiroğlu, Mehmet Can and Güvenir, Halil Altay and Kaya, Kamer},
  title = {Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation Using Knowledge Graphs and Large Language Models},
  journal = {Machine Learning and Knowledge Extraction},
  year = {2026},
  volume = {8},
  number = {5},
  pages = {137},
  doi = {10.3390/make8050137}
}
Start for free →