ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills
PLOS ONE · 24.05.2024 · 19 (5) · e0304013
DOI: 10.1371/journal.pone.0304013 ↗Kurz gefasst
Randomisierte kontrollierte Studie in der Zeitschrift PLOS ONE, online erschienen am 24.05.2024. Verfasst von Zachary A Pardos und 1 weiteren Autor:innen. Beteiligte Einrichtung: Berkeley School of Education, University of California, Berkeley, California, United States of America. Hauptthemen laut MEDLINE: Mathematics, Learning. Die Arbeit wurde bisher 111-mal zitiert (Crossref) und nennt 71 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Peder Sather Center for Advanced Study, Vice Provost of Undergraduate Education, University of California Berkeley, Institute of Cognitive and Brain Sciences, University of California Berkeley.
Abstract (Originalsprache)
Authoring of help content within educational technologies is labor intensive, requiring many iterations of content creation, refining, and proofreading. In this paper, we conduct an efficacy evaluation of ChatGPT-generated help using a 3 x 4 study design (N = 274) to compare the learning gains of ChatGPT to human tutor-authored help across four mathematics problem subject areas. Participants are randomly assigned to one of three hint conditions (control, human tutor, or ChatGPT) paired with one of four randomly assigned subject areas (Elementary Algebra, Intermediate Algebra, College Algebra, or Statistics). We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control, with no statistically significant differences in gains or time-on-task observed between learners receiving ChatGPT vs human tutor help. Notably, ChatGPT-generated help failed quality checks on 32% of problems. This was, however, reducible to nearly 0% for algebra problems and 13% for statistics problems after applying self-consistency, a “hallucination” mitigation technique for Large Language Models.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Berkeley School of Education, University of California, Berkeley, California, United States of America.
- Electrical Engineering and Computer Science, University of California, Berkeley, California, United States of America.
ORCID-Profile: Zachary A Pardos, Shreya Bhandari
Förderung
- Peder Sather Center for Advanced Study
- Vice Provost of Undergraduate Education, University of California Berkeley
- Institute of Cognitive and Brain Sciences, University of California Berkeley
Zitieren
APA
Pardos, Z. A., & Bhandari, S. (2024). ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLOS ONE, 19(5), e0304013. https://doi.org/10.1371/journal.pone.0304013
Vancouver
Pardos ZA, Bhandari S. ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLOS ONE. 2024;19(5):e0304013. doi:10.1371/journal.pone.0304013
BibTeX
@article{Pardos2024,
author = {Pardos, Zachary A and Bhandari, Shreya},
title = {ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills},
journal = {PLOS ONE},
year = {2024},
volume = {19},
number = {5},
pages = {e0304013},
doi = {10.1371/journal.pone.0304013}
}
EUVIMED