EUVIMEDEUVIMED entdecken
Crossref

Auditing the Reliability of Automated Risk Evaluators in Agentic Tutoring Systems Under Pedagogical Style Variations

Wenzhuo Song, Xuechen Wang, Shuang Liang, Jianan Wang

IEEE Transactions on Learning Technologies · 2026 · 19 · 752-765

DOI: 10.1109/tlt.2026.3716609 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Learning Technologies, 2026. Verfasst von Wenzhuo Song und 3 weiteren Autor:innen. Die Arbeit nennt 47 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, Northeast Normal University and Jilin University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Agentic tutoring systems increasingly rely on large language models (LLMs) to generate real-time feedback, yet risky outputs, such as answer leakage, contradiction, and hallucination, can directly affect learning. Answer leakage is a central concern in this setting. It exposes problem solutions to students and can be verified through human annotation. This article presents a pedagogical style-conditioned audit protocol for automated evaluators that detect answer leakage in LLM-generated tutoring feedback before deployment. Using 1100 mathematics problems from the MATH dataset and 4400 feedback items generated under Baseline, Direct, Socratic, and Encouragement styles, we compare rule-based, embedding-based, natural language inference-based, and two LLM-as-a-Judge evaluators from different model families (Qwen2.5-32B and Gemma3-27B) and validate them against a 314-item human-annotated gold standard. The results show that pedagogical style changes how answer-leakage signals appear in feedback and how reliably automated evaluators detect them. Rule-based and LLM-Judge evaluators that perform well under Baseline, Direct, and Encouragement show consistent degradation under Socratic feedback. The primary LLM-Judge (Qwen2.5-32B) achieves overall F1 = 0.726 but drops to 0.427 on Socratic items, and a second LLM-Judge (Gemma3-27B) reproduces this degradation with Socratic F1 = 0.342. The four factorial prompt-sensitivity variants reach overall F1 up to 0.844, but their Socratic F1 remains at or below 0.390. These findings show that automated tutoring-feedback auditing must be style-specific, prompt-calibrated, and human-grounded before deployment.

Förderung

  • National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities
  • Northeast Normal University and Jilin University
  • Social Sciences and Humanities project of the Ministry of Education

Zitieren

APA

Song, W., Wang, X., Liang, S., & Wang, J. (2026). Auditing the Reliability of Automated Risk Evaluators in Agentic Tutoring Systems Under Pedagogical Style Variations. IEEE Transactions on Learning Technologies, 19, 752-765. https://doi.org/10.1109/tlt.2026.3716609

Vancouver

Song W, Wang X, Liang S, Wang J. Auditing the Reliability of Automated Risk Evaluators in Agentic Tutoring Systems Under Pedagogical Style Variations. IEEE Transactions on Learning Technologies. 2026;19:752-765. doi:10.1109/tlt.2026.3716609

BibTeX
@article{Song2026,
  author = {Song, Wenzhuo and Wang, Xuechen and Liang, Shuang and Wang, Jianan},
  title = {Auditing the Reliability of Automated Risk Evaluators in Agentic Tutoring Systems Under Pedagogical Style Variations},
  journal = {IEEE Transactions on Learning Technologies},
  year = {2026},
  volume = {19},
  pages = {752--765},
  doi = {10.1109/tlt.2026.3716609}
}
Kostenlos starten →