EUVIMEDEUVIMED entdecken
Crossref

Automatic Short‐Answer Grading in Sustainability Education: AI –Human Agreement

Emrah Emirtekin, Yasin Özarslan

Journal of Computer Assisted Learning · 2026-02 · 42 (1)

DOI: 10.1002/jcal.70160 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Computer Assisted Learning, online erschienen am 02.12.2025. Verfasst von Emrah Emirtekin und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 54 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Sustainability education emphasises critical thinking and interdisciplinary understanding, making the assessment of students' learning outcomes complex. While Large Language Models (LLMs) have shown promise in educational assessment, their reliability in domains requiring contextual reasoning—such as sustainability—remains unclear.

Objectives

This study aims to evaluate the agreement between human raters and several LLMs (GPT‐4o, Gemini 2.0 Flash, DeepSeek V3, LLaMA 3.3) in assessing short‐answer responses from a university‐level Sustainability course. It also investigates how this agreement varies across cognitive skill levels.

Methods

A total of 232 short‐answer responses were evaluated using a rubric aligned with Bloom's Revised Taxonomy. Consensus scores from human raters were compared to LLM‐generated scores using multiple statistical measures, including Quadratic Weighted Kappa (QWK), Intraclass Correlation Coefficient (ICC), Pearson correlation, and distributional overlap.

Results

Moderate agreement was found between LLMs and human raters in total scores (QWK: 0.585–0.640; r : 0.660–0.668; : 0.681–0.803). Inter‐rater reliability among humans was good to excellent (ICC: 0.667–0.800). Criterion‐level agreement declined as cognitive complexity increased, with notably low agreement on evaluating higher‐order skills.

Conclusions

Overall, LLM–human agreement was moderate on total scores but declined at higher cognitive levels, indicating that LLMs are suitable for basic comprehension checks while human oversight remains necessary for complex reasoning.

Zitieren

APA

Emirtekin, E., & Özarslan, Y. (2026). Automatic Short‐Answer Grading in Sustainability Education: AI –Human Agreement. Journal of Computer Assisted Learning, 42(1). https://doi.org/10.1002/jcal.70160

Vancouver

Emirtekin E, Özarslan Y. Automatic Short‐Answer Grading in Sustainability Education: AI –Human Agreement. Journal of Computer Assisted Learning. 2026;42(1). doi:10.1002/jcal.70160

BibTeX
@article{Emirtekin2026,
  author = {Emirtekin, Emrah and Özarslan, Yasin},
  title = {Automatic Short‐Answer Grading in Sustainability Education: AI –Human Agreement},
  journal = {Journal of Computer Assisted Learning},
  year = {2026},
  volume = {42},
  number = {1},
  doi = {10.1002/jcal.70160}
}
Kostenlos starten →