Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers
Journal of Educational Computing Research · 2024-01 · 61 (8) · 1723-1754
DOI: 10.1177/07356331231191174 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Educational Computing Research, online erschienen am 10.11.2023. Verfasst von Felipe Urrutia und 1 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: Fondo de Fomento al Desarrollo CientÃ-fico y TecnolÃgico.
Abstract (Originalsprache)
Written answers to open-ended questions can have a higher long-term effect on learning than multiple-choice questions. However, it is critical that teachers immediately review the answers, and ask to redo those that are incoherent. This can be a difficult task and can be time-consuming for teachers. A possible solution is to automate the detection of incoherent answers. One option is to automate the review with Large Language Models (LLM). They have a powerful discursive ability that can be used to explain decisions. In this paper, we analyze the responses of fourth graders in mathematics using three LLMs: GPT-3, BLOOM, and YOU. We used them with zero, one, two, three and four shots. We compared their performance with the results of various classifiers trained with Machine Learning (ML). We found that LLMs perform worse than MLs in detecting incoherent answers. The difficulty seems to reside in recursive questions that contain both questions and answers, and in responses from students with typical fourth-grader misspellings. Upon closer examination, we have found that the ChatGPT model faces the same challenges.
Förderung
- Fondo de Fomento al Desarrollo CientÃ-fico y TecnolÃgico
Zitieren
APA
Urrutia, F., & Araya, R. (2024). Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers. Journal of Educational Computing Research, 61(8), 1723-1754. https://doi.org/10.1177/07356331231191174
Vancouver
Urrutia F, Araya R. Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers. Journal of Educational Computing Research. 2024;61(8):1723-1754. doi:10.1177/07356331231191174
BibTeX
@article{Urrutia2024,
author = {Urrutia, Felipe and Araya, Roberto},
title = {Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers},
journal = {Journal of Educational Computing Research},
year = {2024},
volume = {61},
number = {8},
pages = {1723--1754},
doi = {10.1177/07356331231191174}
}
EUVIMED