EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning

Thorben Jansen, Lucas W. Liebenow, Nils-Jonathan Schaller, John Hattie, Jens Möller

Educational Psychology Review · 2026-12 · 38 (1)

DOI: 10.1007/s10648-026-10136-5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Educational Psychology Review, online erschienen am 10.04.2026. Verfasst von Thorben Jansen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 112 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: IPN - Leibniz-Institut für die Pädagogik der Naturwissenschaften und Mathematik.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

By quantitatively integrating results of meta-analyses, second-order meta-analyses (SOMAs) from educational psychology address far-reaching questions of interest to adjacent scientific fields, policy, and educational practice. Conducting a SOMA, however, is time-consuming, particularly the extraction of statistical results from meta-analyses. Large Language Models (LLMs) have the potential to accelerate data extraction, as evidenced by evaluations of data extraction from primary studies. Evaluations of their accuracy in extracting results from meta-analyses to conduct SOMA are needed. We compare the accuracy of three LLMs with that of human experts extracting data from 156 educational meta-analyses investigating students’ achievement, randomly chosen from the Visible Learning database. LLMs and humans extracted effect sizes, the number of included studies, and the number of students. To assess accuracy, we established a gold standard benchmark by resolving discrepancies between LLMs and human experts through expert adjudication. Results show that the distributions of data from all coders and the standard were similar and showed no systematic bias. Accuracy reached ICCs of 0.95/0.81 for the two human experts, and 0.96/0.97/0.96 for LLMs (Gemini2.5Pro/GPT4.1/GPTo3), with percentage agreement of 86%/80% (humans) and 81%/78%/77% (LLMs). The results demonstrate that LLMs achieve data extraction accuracy comparable to that of human experts. We discuss the conditions under which data extracted by LLMs, humans, or a hybrid of both can be considered validated for use in educational SOMAs.

Förderung

  • IPN - Leibniz-Institut für die Pädagogik der Naturwissenschaften und Mathematik

Zitieren

APA

Jansen, T., Liebenow, L. W., Schaller, N., Hattie, J., & Möller, J. (2026). Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning. Educational Psychology Review, 38(1). https://doi.org/10.1007/s10648-026-10136-5

Vancouver

Jansen T, Liebenow LW, Schaller N, Hattie J, Möller J. Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning. Educational Psychology Review. 2026;38(1). doi:10.1007/s10648-026-10136-5

BibTeX
@article{Jansen2026,
  author = {Jansen, Thorben and Liebenow, Lucas W. and Schaller, Nils-Jonathan and Hattie, John and Möller, Jens},
  title = {Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning},
  journal = {Educational Psychology Review},
  year = {2026},
  volume = {38},
  number = {1},
  doi = {10.1007/s10648-026-10136-5}
}
Kostenlos starten →