EUVIMEDObjevte EUVIMED
Systematische ÜbersichtOpen access

Evaluating a Customized Version of ChatGPT for Systematic Review Data Extraction in Health Research: Development and Usability Study

Bewertung einer angepassten Version von ChatGPT für die Datenerhebung in systematischen Reviews in der Gesundheitsforschung: Entwicklungs- und Usability-Studie

Jayden Sercombe iD, Zachary Bryant iD, Jack Wilson iD

JMIR Formative Research · 2025-08-11 · 9 · e68666-e68666

DOI: 10.2196/68666 ↗

Auf einen Blick

Eine angepasste GPT-Version für die Datenerhebung in systematischen Reviews zeigte hohe Übereinstimmung mit menschlichen Reviewer:innen und verkürzte die Bearbeitungszeit deutlich.

Fragestellung

Kann ein custom GPT die Datenerhebung in systematischen Reviews in der Gesundheitsforschung automatisieren?

Methode

Entwicklung und Evaluation eines custom GPT mit OpenAI GPT Builder und RISEN-Framework; 20 Studien aus 2 systematischen Reviews wurden geprüft.

Ergebnisse

Mittlere Zeit für menschliche Datenerhebung: 36 Minuten pro Studie; GPT-Erzeugung: 26,6 Sekunden; anschließende menschliche Prüfung: 13 Minuten. Übereinstimmung: 91,45 % in Review 1 und 89,31 % in Review 2.

Einschränkungen

Weitere Forschung zu Test-Retest-Reliabilität, breiteren Review-Typen und Genauigkeit bei statistischen Daten erforderlich.

Schlussfolgerung der Autor:innen

Das custom GPT reduzierte die Extraktionszeit deutlich und extrahierte Daten mit hoher Genauigkeit, besonders zu Teilnehmer- und Studieneigenschaften.

Kernaussagen

  • Hohe Gesamtübereinstimmung mit menschlichen Reviewer:innen
  • Besonders genau bei Studien- und Teilnehmermerkmalen
  • Schwächer bei methodischen Merkmalen und statistischen Ergebnissen

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Systematische Übersicht in JMIR Formative Research, published online on 2025-08-11. Written by Jayden Sercombe and 2 co-authors. Institution: The Matilda Centre for Research in Mental Health and Substance Use, University of Sydney, Jane Foss Russell Building (G02), Level 6, Sydney, 2006, Au…. Major MEDLINE topics: Systematic Reviews as Topic, Information Storage and Retrieval, Artificial Intelligence. Cited 19 times so far (Crossref); 29 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Systematic reviews are essential for synthesizing research in health sciences; however, they are resource-intensive and prone to human error. The data extraction phase, in which key details of studies are identified and recorded in a systematic manner, may benefit from the application of automation processes. Recent advancements in artificial intelligence, specifically in large language models (LLMs) such as ChatGPT, may streamline this process.

Objective

This study aimed to develop and evaluate a custom Generative Pre-Training Transformer (GPT), named Systematic Review Extractor Pro , for automating the data extraction phase of systematic reviews in health research.

Methods

OpenAI’s GPT Builder was used to create a GPT tailored to extract information from academic manuscripts. The Role, Instruction, Steps, End goal, and Narrowing (RISEN) framework was used to inform prompt engineering for the GPT. A sample of 20 studies from two distinct systematic reviews was used to evaluate the GPT’s performance in extraction. Agreement rates between the GPT outputs and human reviewers were calculated for each study subsection.

Results

The mean time for human data extraction was 36 minutes per study, compared to 26.6 seconds for GPT generation, followed by 13 minutes of human review. The GPT demonstrated high overall agreement rates with human reviewers, achieving 91.45% for review 1 and 89.31% for review 2. It was particularly accurate in extracting study characteristics (review 1: 95.25%; review 2: 90.83%) and participant characteristics (review 1: 95.03%; review 2: 90.00%), with lower performance observed in more complex areas such as methodological characteristics (87.07%) and statistical results (77.50%). The GPT correctly extracted data in 14 instances (3.25% in review 1) and four instances (1.16% in review 2) when the human reviewer was incorrect.

Conclusions

The custom GPT significantly reduced extraction time and shows evidence that it can extract data with high accuracy, particularly for participant and study characteristics. This tool may offer a viable option for researchers seeking to reduce resource demands during the extraction phase, although more research is needed to evaluate test-retest reliability, performance across broader review types, and accuracy in extracting statistical data. The tool developed in the current study has been made open access.

Subjects

MeSH terms (MEDLINE)

Systematic Reviews as TopicInformation Storage and RetrievalArtificial IntelligenceHumansReproducibility of ResultsGenerative Artificial Intelligence

Author keywords

AIChatGPTLLMartificial intelligencedata extractionlarge language modelssystematic reviews

Authors and institutions

  • The Matilda Centre for Research in Mental Health and Substance Use, University of Sydney, Jane Foss Russell Building (G02), Level 6, Sydney, 2006, Australia, 612 8627 9380.

ORCID profiles: Jayden Sercombe, Zachary Bryant, Jack Wilson

Cite

APA

Sercombe, J., Bryant, Z., & Wilson, J. (2025). Evaluating a Customized Version of ChatGPT for Systematic Review Data Extraction in Health Research: Development and Usability Study. JMIR Formative Research, 9, e68666-e68666. https://doi.org/10.2196/68666

Vancouver

Sercombe J, Bryant Z, Wilson J. Evaluating a Customized Version of ChatGPT for Systematic Review Data Extraction in Health Research: Development and Usability Study. JMIR Formative Research. 2025;9:e68666-e68666. doi:10.2196/68666

BibTeX
@article{Sercombe2025,
  author = {Sercombe, Jayden and Bryant, Zachary and Wilson, Jack},
  title = {Evaluating a Customized Version of ChatGPT for Systematic Review Data Extraction in Health Research: Development and Usability Study},
  journal = {JMIR Formative Research},
  year = {2025},
  volume = {9},
  pages = {e68666--e68666},
  doi = {10.2196/68666}
}
Start for free →