EUVIMEDEUVIMED entdecken
Crossref

When Retrieval Can Hurt: A Multi-Role 2×2 Human Evaluation of Retrieval and Self-Refinement for Arabic Multiple-Choice Question Generation

Abderrahman AZI, Abdessamad JARRAR, Moulay AMZIL, Ahmed EL GHAZI

International Journal of Advanced Computer Science and Applications · 2026 · 17 (10)

DOI: 10.14569/ijacsa.2026.0171034 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Advanced Computer Science and Applications, 2026. Verfasst von Abderrahman AZI und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Retrieval-Augmented Generation (RAG) and iterative self-refinement are often added to large language model (LLM) generation pipelines, but their contribution to the quality of Arabic educational items remains uncertain. This study evaluates Modern Standard Arabic multiple-choice question (MCQ) generation in a road-safety domain with a complete 2×2 design crossing retrieval and refinement. One item was generated for each of 31 topics under four conditions, and six raters (students, driving instructors, and licensing examiners) evaluated all 124 items on six criteria, yielding 744 blinded ratings. In a mixed-effects model with crossed random effects for evaluator and topic, retrieval was associated with lower composite ratings (β = −0.94 on a 6–30 scale, 95% CI [−1.45, −0.44], p = .0003; standardized d = −0.28). The decrease was small, about 0.16 Likert points per criterion, but persisted with evaluator-specific random slopes and after adjustment for item length. Question clarity decreased most consistently. Refinement showed no detectable effect (β = −0.07, 95% CI [−0.57, 0.43]) and was statistically equivalent to zero within ±1 composite point. ROUGE-L and BERTScore computed between conditions showed that retrieval-grounded items differed substantially in wording from baseline items, but similarity scores do not indicate the direction of quality change. Two domain experts judged the retrieved passages predominantly relevant and all audited retrieval-plus-refinement answers correct. The findings are bounded to one Llama-family model, one corpus, and one fixed top-k retrieval configuration; they indicate that retrieval should be tested empirically with human evaluation before being adopted by default for Arabic MCQ generation.

Zitieren

APA

AZI, A., JARRAR, A., AMZIL, M., & GHAZI, A. E. (2026). When Retrieval Can Hurt: A Multi-Role 2×2 Human Evaluation of Retrieval and Self-Refinement for Arabic Multiple-Choice Question Generation. International Journal of Advanced Computer Science and Applications, 17(10). https://doi.org/10.14569/ijacsa.2026.0171034

Vancouver

AZI A, JARRAR A, AMZIL M, GHAZI AE. When Retrieval Can Hurt: A Multi-Role 2×2 Human Evaluation of Retrieval and Self-Refinement for Arabic Multiple-Choice Question Generation. International Journal of Advanced Computer Science and Applications. 2026;17(10). doi:10.14569/ijacsa.2026.0171034

BibTeX
@article{AZI2026,
  author = {AZI, Abderrahman and JARRAR, Abdessamad and AMZIL, Moulay and GHAZI, Ahmed EL},
  title = {When Retrieval Can Hurt: A Multi-Role 2×2 Human Evaluation of Retrieval and Self-Refinement for Arabic Multiple-Choice Question Generation},
  journal = {International Journal of Advanced Computer Science and Applications},
  year = {2026},
  volume = {17},
  number = {10},
  doi = {10.14569/ijacsa.2026.0171034}
}
Kostenlos starten →