EUVIMEDEUVIMED entdecken
Crossref

How Consistent Are Humans When Grading Programming Assignments?

Marcus Messer, Neil C. C. Brown, Michael Kölling, Miaojing Shi

ACM Transactions on Computing Education · 31.12.2025 · 25 (4) · 1-37

DOI: 10.1145/3759256 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Computing Education, online erschienen am 18.09.2025. Verfasst von Marcus Messer und 3 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 66 Literaturangaben. Gefördert von: King’s College London College.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Providing consistent summative assessment to students is important, as the grades they are awarded affect their progression through university and future career prospects. While small cohorts are typically assessed by a single assessor, such as the module/class leader, larger cohorts are often assessed by multiple assessors, typically teaching assistants, which increases the risk of inconsistent grading. To investigate the consistency of human grading of programming assignments, we asked 28 participants to each grade 40 CS1 introductory Java assignments, providing grades and feedback for correctness, code elegance, readability and documentation; the 40 assignments were split into two batches of 20. The 28 participants were divided into seven groups of four (where each group graded the same 40 assignments) to allow us to investigate the consistency of a group of assessors. In the second batch of 20, we duplicated one assignment from the first to analyse the internal consistency of individual assessors. We measured the inter-rater reliability of the groups using Krippendorff’s \(\alpha\) —an \(\alpha > 0.667\) is recommended to make tentative conclusions based on the rating. Our groups were inconsistent, with an average \(\alpha=0.2\) when grading correctness and an average \(\alpha < 0.1\) for code elegance, readability and documentation. To measure the individual consistency of graders, we measured the distance between the grades they awarded for the duplicated assignment in batch one and batch two. Only one participant of the 22 who didn’t notice that the assignment was a duplicate was awarded the same grade for correctness, code elegance, readability and documentation. The average grade difference was 1.79 for correctness and less than 1.6 for code elegance, readability and documentation. Our results show that human graders in our study cannot agree on the grade to give a piece of student work and are often individually inconsistent, suggesting that the idea of a ‘gold standard’ of human grading might be flawed. This highlights that a shared rubric alone is not enough to ensure consistency, and other aspects such as assessor training and alternative grading practices should be explored to improve the consistency of human grading further when grading programming assignments.

Förderung

  • King’s College London College

Zitieren

APA

Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2025). How Consistent Are Humans When Grading Programming Assignments?. ACM Transactions on Computing Education, 25(4), 1-37. https://doi.org/10.1145/3759256

Vancouver

Messer M, Brown NCC, Kölling M, Shi M. How Consistent Are Humans When Grading Programming Assignments?. ACM Transactions on Computing Education. 2025;25(4):1-37. doi:10.1145/3759256

BibTeX
@article{Messer2025,
  author = {Messer, Marcus and Brown, Neil C. C. and Kölling, Michael and Shi, Miaojing},
  title = {How Consistent Are Humans When Grading Programming Assignments?},
  journal = {ACM Transactions on Computing Education},
  year = {2025},
  volume = {25},
  number = {4},
  pages = {1--37},
  doi = {10.1145/3759256}
}
Kostenlos starten →