Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning
Educational Psychology Review · 2026-12 · 38 (1)
DOI: 10.1007/s10648-026-10136-5 ↗Kurz gefasst
Fachartikel in der Zeitschrift Educational Psychology Review, online erschienen am 10.04.2026. Verfasst von Thorben Jansen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 112 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: IPN - Leibniz-Institut für die Pädagogik der Naturwissenschaften und Mathematik.
Abstract (Originalsprache)
By quantitatively integrating results of meta-analyses, second-order meta-analyses (SOMAs) from educational psychology address far-reaching questions of interest to adjacent scientific fields, policy, and educational practice. Conducting a SOMA, however, is time-consuming, particularly the extraction of statistical results from meta-analyses. Large Language Models (LLMs) have the potential to accelerate data extraction, as evidenced by evaluations of data extraction from primary studies. Evaluations of their accuracy in extracting results from meta-analyses to conduct SOMA are needed. We compare the accuracy of three LLMs with that of human experts extracting data from 156 educational meta-analyses investigating students’ achievement, randomly chosen from the Visible Learning database. LLMs and humans extracted effect sizes, the number of included studies, and the number of students. To assess accuracy, we established a gold standard benchmark by resolving discrepancies between LLMs and human experts through expert adjudication. Results show that the distributions of data from all coders and the standard were similar and showed no systematic bias. Accuracy reached ICCs of 0.95/0.81 for the two human experts, and 0.96/0.97/0.96 for LLMs (Gemini2.5Pro/GPT4.1/GPTo3), with percentage agreement of 86%/80% (humans) and 81%/78%/77% (LLMs). The results demonstrate that LLMs achieve data extraction accuracy comparable to that of human experts. We discuss the conditions under which data extracted by LLMs, humans, or a hybrid of both can be considered validated for use in educational SOMAs.
Förderung
- IPN - Leibniz-Institut für die Pädagogik der Naturwissenschaften und Mathematik
Zitieren
APA
Jansen, T., Liebenow, L. W., Schaller, N., Hattie, J., & Möller, J. (2026). Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning. Educational Psychology Review, 38(1). https://doi.org/10.1007/s10648-026-10136-5
Vancouver
Jansen T, Liebenow LW, Schaller N, Hattie J, Möller J. Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning. Educational Psychology Review. 2026;38(1). doi:10.1007/s10648-026-10136-5
BibTeX
@article{Jansen2026,
author = {Jansen, Thorben and Liebenow, Lucas W. and Schaller, Nils-Jonathan and Hattie, John and Möller, Jens},
title = {Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning},
journal = {Educational Psychology Review},
year = {2026},
volume = {38},
number = {1},
doi = {10.1007/s10648-026-10136-5}
}
EUVIMED