ChatGPT-4o in Risk-of-Bias Assessments in Neonatology: A Validity Analysis
Neonatology · 25.02.2025 · 122 (3) · 360-365
DOI: 10.1159/000544857 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neonatology, online erschienen am 25.02.2025. Verfasst von Ilari Kuitunen und 2 weiteren Autor:innen. Beteiligte Einrichtung: Kuopio Pediatric Research Unit, University of Eastern Finland, Kuopio, Finland. Hauptthemen laut MEDLINE: Neonatology, Language. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Introduction
Only a few studies have addressed the potential of large language models (LLMs) in risk-of-bias assessments and the results have been varying. The aim of this study was to analyze how well ChatGPT performs in risk-of-bias assessments of neonatal studies.
Methods
We searched all Cochrane neonatal intervention reviews published in 2024 and extracted all risk-of-bias assessments. Then the full reports were retrieved and uploaded alongside the guidance to perform a Cochrane original risk-of-bias analysis in ChatGPT-4o. The concordance between the original assessment and that provided by ChatGPT-4o was evaluated by inter-class correlation coefficients and Cohen’s kappa statistics (with 95% confidence intervals) for each risk-of-bias domain and for the overall assessment.
Results
From 9 reviews, a total of 61 randomized studies were analyzed. A total of 427 judgments were compared. The overall κ was 0.43 (95% CI: 0.35–0.51) and the overall intraclass correlation coefficient was 0.65 (95% CI: 0.59–0.70). The Cohen’s κ was assessed for each domain and the best agreement was observed in the allocation concealment (κ = 0.73, 95% CI: 0.55–0.90), whereas the poorest agreement was found in incomplete outcome data (κ = −0.03, 95% CI: −0.07–0.02).
Conclusion
ChatGPT-4o failed to achieve sufficient agreement in the risk-of-bias assessments. Future studies should examine whether the performance of other LLM would be better or whether the agreement in ChatGPT-4o could be further enhanced by better prompting. Currently, the use of ChatGPT-4o in risk-of-bias assessments should not be promoted.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Kuopio Pediatric Research Unit, University of Eastern Finland, Kuopio, Finland.
- Department of Pediatrics and Neonatology, Kuopio University Hospital, Kuopio, Finland.
- Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland.
- Center for Musculoskeletal Disease, Tampere University Hospital, Tampere, Finland.
- Division of Pediatrics and Neonatal critical Care, "A.Beclere" Hospital, APHP-Paris Saclay University, Paris, France.
- Physiopathology and therapeutic innovation Unit-INSERM U999, Paris Saclay University, Paris, France.
Zitieren
APA
Kuitunen, I., Nyrhi, L., & De Luca, D. (2025). ChatGPT-4o in Risk-of-Bias Assessments in Neonatology: A Validity Analysis. Neonatology, 122(3), 360-365. https://doi.org/10.1159/000544857
Vancouver
Kuitunen I, Nyrhi L, De Luca D. ChatGPT-4o in Risk-of-Bias Assessments in Neonatology: A Validity Analysis. Neonatology. 2025;122(3):360-365. doi:10.1159/000544857
BibTeX
@article{Kuitunen2025,
author = {Kuitunen, Ilari and Nyrhi, Lauri and De Luca, Daniele},
title = {ChatGPT-4o in Risk-of-Bias Assessments in Neonatology: A Validity Analysis},
journal = {Neonatology},
year = {2025},
volume = {122},
number = {3},
pages = {360--365},
doi = {10.1159/000544857}
}
EUVIMED