EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

ChatGPT-4o in Risk-of-Bias Assessments in Neonatology: A Validity Analysis

Ilari Kuitunen, Lauri Nyrhi, Daniele De Luca

Neonatology · 25.02.2025 · 122 (3) · 360-365

DOI: 10.1159/000544857 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Neonatology, online erschienen am 25.02.2025. Verfasst von Ilari Kuitunen und 2 weiteren Autor:innen. Beteiligte Einrichtung: Kuopio Pediatric Research Unit, University of Eastern Finland, Kuopio, Finland. Hauptthemen laut MEDLINE: Neonatology, Language. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Only a few studies have addressed the potential of large language models (LLMs) in risk-of-bias assessments and the results have been varying. The aim of this study was to analyze how well ChatGPT performs in risk-of-bias assessments of neonatal studies.

Methods

We searched all Cochrane neonatal intervention reviews published in 2024 and extracted all risk-of-bias assessments. Then the full reports were retrieved and uploaded alongside the guidance to perform a Cochrane original risk-of-bias analysis in ChatGPT-4o. The concordance between the original assessment and that provided by ChatGPT-4o was evaluated by inter-class correlation coefficients and Cohen’s kappa statistics (with 95% confidence intervals) for each risk-of-bias domain and for the overall assessment.

Results

From 9 reviews, a total of 61 randomized studies were analyzed. A total of 427 judgments were compared. The overall κ was 0.43 (95% CI: 0.35–0.51) and the overall intraclass correlation coefficient was 0.65 (95% CI: 0.59–0.70). The Cohen’s κ was assessed for each domain and the best agreement was observed in the allocation concealment (κ = 0.73, 95% CI: 0.55–0.90), whereas the poorest agreement was found in incomplete outcome data (κ = −0.03, 95% CI: −0.07–0.02).

Conclusion

ChatGPT-4o failed to achieve sufficient agreement in the risk-of-bias assessments. Future studies should examine whether the performance of other LLM would be better or whether the agreement in ChatGPT-4o could be further enhanced by better prompting. Currently, the use of ChatGPT-4o in risk-of-bias assessments should not be promoted.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

NeonatologyLanguageHumansInfant, NewbornRisk AssessmentBiasReproducibility of ResultsRandomized Controlled Trials as TopicGenerative Artificial Intelligence

Stichwörter der Autor:innen

CochraneEvidence synthesisRisk of bias

Autor:innen und Institutionen

  • Kuopio Pediatric Research Unit, University of Eastern Finland, Kuopio, Finland.
  • Department of Pediatrics and Neonatology, Kuopio University Hospital, Kuopio, Finland.
  • Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland.
  • Center for Musculoskeletal Disease, Tampere University Hospital, Tampere, Finland.
  • Division of Pediatrics and Neonatal critical Care, "A.Beclere" Hospital, APHP-Paris Saclay University, Paris, France.
  • Physiopathology and therapeutic innovation Unit-INSERM U999, Paris Saclay University, Paris, France.

Zitieren

APA

Kuitunen, I., Nyrhi, L., & De Luca, D. (2025). ChatGPT-4o in Risk-of-Bias Assessments in Neonatology: A Validity Analysis. Neonatology, 122(3), 360-365. https://doi.org/10.1159/000544857

Vancouver

Kuitunen I, Nyrhi L, De Luca D. ChatGPT-4o in Risk-of-Bias Assessments in Neonatology: A Validity Analysis. Neonatology. 2025;122(3):360-365. doi:10.1159/000544857

BibTeX
@article{Kuitunen2025,
  author = {Kuitunen, Ilari and Nyrhi, Lauri and De Luca, Daniele},
  title = {ChatGPT-4o in Risk-of-Bias Assessments in Neonatology: A Validity Analysis},
  journal = {Neonatology},
  year = {2025},
  volume = {122},
  number = {3},
  pages = {360--365},
  doi = {10.1159/000544857}
}
Kostenlos starten →