EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Automating the data extraction process for systematic reviews using GPT-4o and o3

Yuki Kataoka, Tomohiro Takayama, Keisuke Yoshimura iD, Ryuhei So, Yasushi Tsujimoto, Yosuke Yamagishi, Shiro Takagi, Yuki Furukawa, Masatsugu Sakata, Đorđe Bašić, Andrea Cipriani, Pim Cuijpers und 8 weitere

Research Synthesis Methods · 2026-01 · 17 (1) · 42-62

DOI: 10.1017/rsm.2025.10030 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research Synthesis Methods, online erschienen am 17.09.2025. Verfasst von Yuki Kataoka und 19 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine, Kyoto Min-iren Asukai Hospital, Kyoto, Japan. Hauptthemen laut MEDLINE: Generative Artificial Intelligence, Information Storage and Retrieval, Systematic Reviews as Topic. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models have shown promise for automating data extraction (DE) in systematic reviews (SRs), but most existing approaches require manual interaction. We developed an open-source system using GPT-4o to automatically extract data with no human intervention during the extraction process. We developed the system on a dataset of 290 randomized controlled trials (RCTs) from a published SR about cognitive behavioral therapy for insomnia. We evaluated the system on two other datasets: 5 RCTs from an updated search for the same review and 10 RCTs used in a separate published study that had also evaluated automated DE. We developed the best approach across all variables in the development dataset using GPT-4o. The performance in the updated-search dataset using o3 was 74.9% sensitivity, 76.7% specificity, 75.7 precision, 93.5% variable detection comprehensiveness, and 75.3% accuracy. In both datasets, accuracy was higher for string variables (e.g., country, study design, drug names, and outcome definitions) compared with numeric variables. In the third external validation dataset, GPT-4o showed a lower performance with a mean accuracy of 84.4% compared with the previous study. However, by adjusting our DE method, while maintaining the same prompting technique, we achieved a mean accuracy of 96.3%, which was comparable to the previous manual extraction study. Our system shows potential for assisting the DE of string variables alongside a human reviewer. However, it cannot yet replace humans for numeric DE. Further evaluation across diverse review contexts is needed to establish broader applicability.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Generative Artificial IntelligenceInformation Storage and RetrievalSystematic Reviews as TopicHumansAutomationLarge Language ModelsRandomized Controlled Trials as TopicReproducibility of ResultsSensitivity and SpecificitySoftware

Stichwörter der Autor:innen

GPT-4odata extraction automationlarge language modelso3systematic reviews

Autor:innen und Institutionen

  • Department of Internal Medicine, Kyoto Min-iren Asukai Hospital, Kyoto, Japan.
  • Scientific Research WorkS Peer Support Group (SRWS-PSG), Osaka, Japan.
  • Department of Healthcare Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan.
  • Department of International and Community Oral Health, Tohoku University Graduate School of Dentistry, Sendai, Japan.
  • Faculty of Medicine, Kyoto University, Kyoto, Japan.
  • Fitting Cloud Inc., Kyoto, Japan.

ORCID-Profile: Keisuke Yoshimura

Förderung

  • Japan Society for the Promotion of Science

Zitieren

APA

Kataoka, Y., Takayama, T., Yoshimura, K., So, R., Tsujimoto, Y., Yamagishi, Y., Takagi, S., Furukawa, Y., Sakata, M., Bašić, Đ., Cipriani, A., Cuijpers, P., Karyotaki, E., Harrer, M., Leucht, S., Homiar, A., Ostinelli, E. G., Miguel, C., Rodolico, A., & Furukawa, T. A. (2026). Automating the data extraction process for systematic reviews using GPT-4o and o3. Research Synthesis Methods, 17(1), 42-62. https://doi.org/10.1017/rsm.2025.10030

Vancouver

Kataoka Y, Takayama T, Yoshimura K, So R, Tsujimoto Y, Yamagishi Y, et al. Automating the data extraction process for systematic reviews using GPT-4o and o3. Research Synthesis Methods. 2026;17(1):42-62. doi:10.1017/rsm.2025.10030

BibTeX
@article{Kataoka2026,
  author = {Kataoka, Yuki and Takayama, Tomohiro and Yoshimura, Keisuke and So, Ryuhei and Tsujimoto, Yasushi and Yamagishi, Yosuke and Takagi, Shiro and Furukawa, Yuki and Sakata, Masatsugu and Bašić, Đorđe and Cipriani, Andrea and Cuijpers, Pim and Karyotaki, Eirini and Harrer, Mathias and Leucht, Stefan and Homiar, Ava and Ostinelli, Edoardo G and Miguel, Clara and Rodolico, Alessandro and Furukawa, Toshi A},
  title = {Automating the data extraction process for systematic reviews using GPT-4o and o3},
  journal = {Research Synthesis Methods},
  year = {2026},
  volume = {17},
  number = {1},
  pages = {42--62},
  doi = {10.1017/rsm.2025.10030}
}
Kostenlos starten →