When Even Experts Disagree: Human-in-the-Loop Evaluation of LLM-Generated Hypotheses for Learning Analytics
IEEE Access · 2026 · 14 · 106842-106860
DOI: 10.1109/access.2026.3708316 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Kyosuke Takami und 2 weiteren Autor:innen. Die Arbeit nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science (JSPS) KAKENHI Grant-in-Aid for Early-Career Scientists, JSPS KAKENHI Grant-in-Aid for Scientific Research.
Abstract (Originalsprache)
Hypothesis formulation is central to learning analytics (LA), yet how to appraise hypotheses generated by large language models (LLMs) remains unclear. This study constructs a domain-specific dataset from 992 papers published at the International Conference on Learning Analytics and Knowledge (LAK) between 2011 and 2024 and distills 963 keyword–hypothesis pairs to examine whether LLMs support early-stage ideation and how reliably humans judge their outputs. Across 15 topics and 495 pairwise evaluations by LA experts (n= 5) and K–12 practitioners (n= 6), fine-tuning provides no uniform advantage: experts are nearly split between fine-tuned GPT-4o and non-fine-tuned GPT-4o (52% vs. 48%), while practitioners prefer non-fine-tuned models (up to 63%). Inter-rater agreement was low (Fleiss’ $\kappa $ : −0.014 for experts; 0.21 for practitioners), and practitioners reported greater decision difficulty. Bradley–Terry ranking placed the non-fine-tuned GPT-4o-mini highest, showing modest advantages over the other models in pairwise comparisons. Criterion-selection analyses indicate that experts prioritize specificity, impact, and clarity, whereas practitioners emphasize concreteness, usefulness, and conciseness. Overall, the findings suggest that evaluation—rather than generation—is the rate-limiting step for LLM-assisted hypothesis work in education, and highlight audience-targeted prompting and lightweight, rationale-backed pairwise appraisal as pragmatic scaffolds for producing decision-ready hypotheses with implications for AI for Science.
Förderung
- Japan Society for the Promotion of Science (JSPS) KAKENHI Grant-in-Aid for Early-Career Scientists
- JSPS KAKENHI Grant-in-Aid for Scientific Research
Zitieren
APA
Takami, K., Majumdar, R., & Flanagan, B. (2026). When Even Experts Disagree: Human-in-the-Loop Evaluation of LLM-Generated Hypotheses for Learning Analytics. IEEE Access, 14, 106842-106860. https://doi.org/10.1109/access.2026.3708316
Vancouver
Takami K, Majumdar R, Flanagan B. When Even Experts Disagree: Human-in-the-Loop Evaluation of LLM-Generated Hypotheses for Learning Analytics. IEEE Access. 2026;14:106842-106860. doi:10.1109/access.2026.3708316
BibTeX
@article{Takami2026,
author = {Takami, Kyosuke and Majumdar, Rwitajit and Flanagan, Brendan},
title = {When Even Experts Disagree: Human-in-the-Loop Evaluation of LLM-Generated Hypotheses for Learning Analytics},
journal = {IEEE Access},
year = {2026},
volume = {14},
pages = {106842--106860},
doi = {10.1109/access.2026.3708316}
}
EUVIMED