EUVIMEDEUVIMED entdecken
ValidierungsstudieVergleichsstudieOpen Access · CC BY 4.0

Large language models are comparable with commonly used statistical software: A validation of GPT 5.1 for frequentist meta‐analysis in orthopaedics

Mikhail Salzmann iD, Nikolai Ramadanov, Robert Prill, Robert Hable, Roland Becker

Knee Surgery, Sports Traumatology, Arthroscopy · 2026

DOI: 10.1002/ksa.70379 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Knee Surgery, Sports Traumatology, Arthroscopy, 2026. Verfasst von Mikhail Salzmann und 4 weiteren Autor:innen. Beteiligte Einrichtung: Centre of Orthopaedics, Traumatology and Plastic Surgery, Brandenburg Medical School, University Hospital Brandenburg an der Havel, Brandenburg an de…. Hauptthemen laut MEDLINE: Large Language Models, Meta-Analysis as Topic, Software, Orthopedics. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

The purpose of this study was to evaluate whether Chat Generative Pre‐trained Transformer (ChatGPT; Version 5.1) can reproduce frequentist meta‐analytic calculations with an accuracy comparable to established statistical software in orthopaedic research.

Methods

In this methodological comparison study, data from two previously published orthopaedic meta‐analyses with identical statistical architectures as reference standards were used. Between‐study variance ( τ 2 ) was estimated using the Sidik–Jonkman method and uncertainty was quantified using the Hartung–Knapp adjustment for the random‐effects models, while common‐effect models assume τ 2 = 0. Original data extraction tables were provided to ChatGPT‐5.1, which was instructed to perform the same analyses. ChatGPT‐generated pooled mean differences, confidence intervals and heterogeneity statistics ( I 2 , τ 2 , p values) were compared with verified reference results obtained using the meta and metafor packages in R.

Results

Across seven evaluated outcomes, ChatGPT‐5.1 reproduced the direction of effects in all cases. Deviations compared with reference meta‐analyses were classified as minor in three outcomes (43%), moderate in one outcome (14%) and major in three outcomes (43%). Agreement was highest in low‐heterogeneity settings, whereas substantial deviations occurred in outcomes with pronounced between‐study heterogeneity, particularly under random‐effects models.

Conclusion

ChatGPT‐5.1 demonstrates emerging capability to approximate frequentist meta‐analytic calculations, particularly in low‐heterogeneity settings. However, its tendency to underestimate between‐study variability and to deviate in complex random‐effects scenarios limits its reliability as a standalone tool. At present, large language models may support exploratory analyses but cannot fully replace dedicated statistical software for meta‐analyses in orthopaedic research. Level of Evidence Level III.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsMeta-Analysis as TopicSoftwareOrthopedicsHumansGenerative Artificial IntelligenceReproducibility of Results

Stichwörter der Autor:innen

artificial intelligenceheterogeneitymeta‐researchorthopaedic researchstatistical validation

Autor:innen und Institutionen

  • Centre of Orthopaedics, Traumatology and Plastic Surgery, Brandenburg Medical School, University Hospital Brandenburg an der Havel, Brandenburg an der Havel, Germany.
  • Faculty of Health Science Brandenburg, Brandenburg Medical School Theodor Fontane, Brandenburg an der Havel, Germany.
  • Faculty of Applied Computer Science, Deggendorf Institute of Technology, Deggendorf, Germany.

ORCID-Profile: Mikhail Salzmann

Zitieren

APA

Salzmann, M., Ramadanov, N., Prill, R., Hable, R., & Becker, R. (2026). Large language models are comparable with commonly used statistical software: A validation of GPT 5.1 for frequentist meta‐analysis in orthopaedics. Knee Surgery, Sports Traumatology, Arthroscopy. https://doi.org/10.1002/ksa.70379

Vancouver

Salzmann M, Ramadanov N, Prill R, Hable R, Becker R. Large language models are comparable with commonly used statistical software: A validation of GPT 5.1 for frequentist meta‐analysis in orthopaedics. Knee Surgery, Sports Traumatology, Arthroscopy. 2026. doi:10.1002/ksa.70379

BibTeX
@article{Salzmann2026,
  author = {Salzmann, Mikhail and Ramadanov, Nikolai and Prill, Robert and Hable, Robert and Becker, Roland},
  title = {Large language models are comparable with commonly used statistical software: A validation of GPT 5.1 for frequentist meta‐analysis in orthopaedics},
  journal = {Knee Surgery, Sports Traumatology, Arthroscopy},
  year = {2026},
  doi = {10.1002/ksa.70379}
}
Kostenlos starten →