Automating the data extraction process for systematic reviews using GPT-4o and o3
Research Synthesis Methods · 2026-01 · 17 (1) · 42-62
DOI: 10.1017/rsm.2025.10030 ↗Kurz gefasst
Fachartikel in der Zeitschrift Research Synthesis Methods, online erschienen am 17.09.2025. Verfasst von Yuki Kataoka und 19 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine, Kyoto Min-iren Asukai Hospital, Kyoto, Japan. Hauptthemen laut MEDLINE: Generative Artificial Intelligence, Information Storage and Retrieval, Systematic Reviews as Topic. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science.
Abstract (Originalsprache)
Large language models have shown promise for automating data extraction (DE) in systematic reviews (SRs), but most existing approaches require manual interaction. We developed an open-source system using GPT-4o to automatically extract data with no human intervention during the extraction process. We developed the system on a dataset of 290 randomized controlled trials (RCTs) from a published SR about cognitive behavioral therapy for insomnia. We evaluated the system on two other datasets: 5 RCTs from an updated search for the same review and 10 RCTs used in a separate published study that had also evaluated automated DE. We developed the best approach across all variables in the development dataset using GPT-4o. The performance in the updated-search dataset using o3 was 74.9% sensitivity, 76.7% specificity, 75.7 precision, 93.5% variable detection comprehensiveness, and 75.3% accuracy. In both datasets, accuracy was higher for string variables (e.g., country, study design, drug names, and outcome definitions) compared with numeric variables. In the third external validation dataset, GPT-4o showed a lower performance with a mean accuracy of 84.4% compared with the previous study. However, by adjusting our DE method, while maintaining the same prompting technique, we achieved a mean accuracy of 96.3%, which was comparable to the previous manual extraction study. Our system shows potential for assisting the DE of string variables alongside a human reviewer. However, it cannot yet replace humans for numeric DE. Further evaluation across diverse review contexts is needed to establish broader applicability.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Internal Medicine, Kyoto Min-iren Asukai Hospital, Kyoto, Japan.
- Scientific Research WorkS Peer Support Group (SRWS-PSG), Osaka, Japan.
- Department of Healthcare Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan.
- Department of International and Community Oral Health, Tohoku University Graduate School of Dentistry, Sendai, Japan.
- Faculty of Medicine, Kyoto University, Kyoto, Japan.
- Fitting Cloud Inc., Kyoto, Japan.
ORCID-Profile: Keisuke Yoshimura
Förderung
- Japan Society for the Promotion of Science
Zitieren
APA
Kataoka, Y., Takayama, T., Yoshimura, K., So, R., Tsujimoto, Y., Yamagishi, Y., Takagi, S., Furukawa, Y., Sakata, M., Bašić, Đ., Cipriani, A., Cuijpers, P., Karyotaki, E., Harrer, M., Leucht, S., Homiar, A., Ostinelli, E. G., Miguel, C., Rodolico, A., & Furukawa, T. A. (2026). Automating the data extraction process for systematic reviews using GPT-4o and o3. Research Synthesis Methods, 17(1), 42-62. https://doi.org/10.1017/rsm.2025.10030
Vancouver
Kataoka Y, Takayama T, Yoshimura K, So R, Tsujimoto Y, Yamagishi Y, et al. Automating the data extraction process for systematic reviews using GPT-4o and o3. Research Synthesis Methods. 2026;17(1):42-62. doi:10.1017/rsm.2025.10030
BibTeX
@article{Kataoka2026,
author = {Kataoka, Yuki and Takayama, Tomohiro and Yoshimura, Keisuke and So, Ryuhei and Tsujimoto, Yasushi and Yamagishi, Yosuke and Takagi, Shiro and Furukawa, Yuki and Sakata, Masatsugu and Bašić, Đorđe and Cipriani, Andrea and Cuijpers, Pim and Karyotaki, Eirini and Harrer, Mathias and Leucht, Stefan and Homiar, Ava and Ostinelli, Edoardo G and Miguel, Clara and Rodolico, Alessandro and Furukawa, Toshi A},
title = {Automating the data extraction process for systematic reviews using GPT-4o and o3},
journal = {Research Synthesis Methods},
year = {2026},
volume = {17},
number = {1},
pages = {42--62},
doi = {10.1017/rsm.2025.10030}
}
EUVIMED