EUVIMEDEUVIMED entdecken
Crossref

Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers

Felipe Urrutia, Roberto Araya

Journal of Educational Computing Research · 2024-01 · 61 (8) · 1723-1754

DOI: 10.1177/07356331231191174 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Educational Computing Research, online erschienen am 10.11.2023. Verfasst von Felipe Urrutia und 1 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: Fondo de Fomento al Desarrollo CientÃ-fico y TecnolÃgico.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Written answers to open-ended questions can have a higher long-term effect on learning than multiple-choice questions. However, it is critical that teachers immediately review the answers, and ask to redo those that are incoherent. This can be a difficult task and can be time-consuming for teachers. A possible solution is to automate the detection of incoherent answers. One option is to automate the review with Large Language Models (LLM). They have a powerful discursive ability that can be used to explain decisions. In this paper, we analyze the responses of fourth graders in mathematics using three LLMs: GPT-3, BLOOM, and YOU. We used them with zero, one, two, three and four shots. We compared their performance with the results of various classifiers trained with Machine Learning (ML). We found that LLMs perform worse than MLs in detecting incoherent answers. The difficulty seems to reside in recursive questions that contain both questions and answers, and in responses from students with typical fourth-grader misspellings. Upon closer examination, we have found that the ChatGPT model faces the same challenges.

Förderung

  • Fondo de Fomento al Desarrollo CientÃ-fico y TecnolÃgico

Zitieren

APA

Urrutia, F., & Araya, R. (2024). Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers. Journal of Educational Computing Research, 61(8), 1723-1754. https://doi.org/10.1177/07356331231191174

Vancouver

Urrutia F, Araya R. Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers. Journal of Educational Computing Research. 2024;61(8):1723-1754. doi:10.1177/07356331231191174

BibTeX
@article{Urrutia2024,
  author = {Urrutia, Felipe and Araya, Roberto},
  title = {Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers},
  journal = {Journal of Educational Computing Research},
  year = {2024},
  volume = {61},
  number = {8},
  pages = {1723--1754},
  doi = {10.1177/07356331231191174}
}
Kostenlos starten →