Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy
Applied Sciences · 11.01.2025 · 15 (2) · 671
DOI: 10.3390/app15020671 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 11.01.2025. Verfasst von Hyein Seo und 6 weiteren Autor:innen. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT), National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT), Chungnam National University.
Abstract (Originalsprache)
The recent advancements in large language models (LLMs) have brought significant changes to the field of education, particularly in the generation and evaluation of feedback. LLMs are transforming education by streamlining tasks like content creation, feedback generation, and assessment, reducing teachers’ workload and improving online education efficiency. This study aimed to verify the consistency and reliability of LLMs as evaluators by conducting automated evaluations using various LLMs based on five educational evaluation criteria. The analysis revealed that while LLMs were capable of performing consistent evaluations under certain conditions, a lack of consistency was observed both among evaluators and across models for other criteria. Notably, low agreement among human evaluators correlated with reduced reliability in LLM evaluations. Furthermore, variations in evaluation results were influenced by factors such as prompt strategies and model architecture, highlighting the complexity of achieving reliable assessments using LLMs. These findings suggest that while LLMs have the potential to transform educational systems, careful selection and combination of models are essential to improve their consistency and align their performance with human evaluators in educational settings.
Förderung
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)
- National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
- Chungnam National University
Zitieren
APA
Seo, H., Hwang, T., Jung, J., Kang, H., Namgoong, H., Lee, Y., & Jung, S. (2025). Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy. Applied Sciences, 15(2), 671. https://doi.org/10.3390/app15020671
Vancouver
Seo H, Hwang T, Jung J, Kang H, Namgoong H, Lee Y, et al. Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy. Applied Sciences. 2025;15(2):671. doi:10.3390/app15020671
BibTeX
@article{Seo2025,
author = {Seo, Hyein and Hwang, Taewook and Jung, Jeesu and Kang, Hyeonseok and Namgoong, Hyuk and Lee, Yohan and Jung, Sangkeun},
title = {Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy},
journal = {Applied Sciences},
year = {2025},
volume = {15},
number = {2},
pages = {671},
doi = {10.3390/app15020671}
}
EUVIMED