EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

When Even Experts Disagree: Human-in-the-Loop Evaluation of LLM-Generated Hypotheses for Learning Analytics

Kyosuke Takami, Rwitajit Majumdar, Brendan Flanagan

IEEE Access · 2026 · 14 · 106842-106860

DOI: 10.1109/access.2026.3708316 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Kyosuke Takami und 2 weiteren Autor:innen. Die Arbeit nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science (JSPS) KAKENHI Grant-in-Aid for Early-Career Scientists, JSPS KAKENHI Grant-in-Aid for Scientific Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Hypothesis formulation is central to learning analytics (LA), yet how to appraise hypotheses generated by large language models (LLMs) remains unclear. This study constructs a domain-specific dataset from 992 papers published at the International Conference on Learning Analytics and Knowledge (LAK) between 2011 and 2024 and distills 963 keyword–hypothesis pairs to examine whether LLMs support early-stage ideation and how reliably humans judge their outputs. Across 15 topics and 495 pairwise evaluations by LA experts (n= 5) and K–12 practitioners (n= 6), fine-tuning provides no uniform advantage: experts are nearly split between fine-tuned GPT-4o and non-fine-tuned GPT-4o (52% vs. 48%), while practitioners prefer non-fine-tuned models (up to 63%). Inter-rater agreement was low (Fleiss’ $\kappa $ : −0.014 for experts; 0.21 for practitioners), and practitioners reported greater decision difficulty. Bradley–Terry ranking placed the non-fine-tuned GPT-4o-mini highest, showing modest advantages over the other models in pairwise comparisons. Criterion-selection analyses indicate that experts prioritize specificity, impact, and clarity, whereas practitioners emphasize concreteness, usefulness, and conciseness. Overall, the findings suggest that evaluation—rather than generation—is the rate-limiting step for LLM-assisted hypothesis work in education, and highlight audience-targeted prompting and lightweight, rationale-backed pairwise appraisal as pragmatic scaffolds for producing decision-ready hypotheses with implications for AI for Science.

Förderung

  • Japan Society for the Promotion of Science (JSPS) KAKENHI Grant-in-Aid for Early-Career Scientists
  • JSPS KAKENHI Grant-in-Aid for Scientific Research

Zitieren

APA

Takami, K., Majumdar, R., & Flanagan, B. (2026). When Even Experts Disagree: Human-in-the-Loop Evaluation of LLM-Generated Hypotheses for Learning Analytics. IEEE Access, 14, 106842-106860. https://doi.org/10.1109/access.2026.3708316

Vancouver

Takami K, Majumdar R, Flanagan B. When Even Experts Disagree: Human-in-the-Loop Evaluation of LLM-Generated Hypotheses for Learning Analytics. IEEE Access. 2026;14:106842-106860. doi:10.1109/access.2026.3708316

BibTeX
@article{Takami2026,
  author = {Takami, Kyosuke and Majumdar, Rwitajit and Flanagan, Brendan},
  title = {When Even Experts Disagree: Human-in-the-Loop Evaluation of LLM-Generated Hypotheses for Learning Analytics},
  journal = {IEEE Access},
  year = {2026},
  volume = {14},
  pages = {106842--106860},
  doi = {10.1109/access.2026.3708316}
}
Kostenlos starten →