EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus‐based bone stress injury

Silvia Gandini iD, Jakob Lindskog, Yinan Yu, Felix Öttl, Richard Willy iD, Paolo Emilio Adami, Francesco Della Villa, Stefano Di Paolo, Fabio Bonsanto, Stefano Mazzoni, Francesco Feletti, Kristian Samuelsson und 1 weitere

Journal of Experimental Orthopaedics · 2026-10 · 13 (4)

DOI: 10.1002/jeo2.70939 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Experimental Orthopaedics, online erschienen am 07.10.2026. Verfasst von Silvia Gandini und 12 weiteren Autor:innen. Beteiligte Einrichtung: Milan Lab Research Department AC Milan S.p.A Milan Italy. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

This study evaluated five publicly available large language model‐based systems using a multidimensional, expert‐based assessment of consensus‐based clinical questions on bone stress injuries.

Methods

Ten standardised questions derived from a 2025 international Delphi consensus on bone stress injuries were submitted to ChatGPT‐5, Perplexity AI, Claude 4, Google Gemini 2.5 and Microsoft Copilot. Responses were independently rated by clinical experts using a multidimensional evaluation framework covering accuracy, comprehensiveness, understanding, reasoning, clarity, potential harm and trust, each rated on a 5‐point Likert scale. Median scores were calculated for each question and model, and comparisons were performed using Kruskal–Wallis tests followed by Dunn's post hoc tests with Bonferroni correction. Interrater agreement was assessed using the weighted Gwet's agreement coefficient. Reproducibility was assessed by repeatedly submitting nine questions representing no, partial, and full expert consensus to each model, with response consistency scored from 1 to 3.

Results

Significant differences between models were observed for comprehensiveness, reasoning, clarity and trust (all p < 0.001), with no differences in accuracy or harm. ChatGPT‐5 and Claude 4 achieved significantly higher comprehensiveness scores, while ChatGPT‐5 and Google Gemini 2.5 had significantly higher reasoning scores than Microsoft Copilot. Claude 4 demonstrated significantly lower clarity than all other models. ChatGPT‐5 achieved significantly higher trust scores and showed the highest mean reproducibility score, followed by Perplexity AI and Gemini 2.5; Microsoft Copilot showed the lowest reproducibility. Interrater agreement ranged from fair to almost perfect, with substantial overall agreement.

Conclusions

Publicly available large language model‐based systems demonstrated moderate‐to‐high performance in addressing expert‐derived, consensus‐based questions on bone stress injuries, suggesting potential to support the communication of consensus‐based knowledge. However, performance varied across evaluation dimensions and reproducibility, indicating that large language models may support, but should not replace, primary consensus documents or expert clinical judgement. Level of Evidence N/A.

Schlagwörter

Stichwörter der Autor:innen

Delphi consensusartificial intelligencebone stress injurieslarge language modelssports medicine

Autor:innen und Institutionen

  • Milan Lab Research Department AC Milan S.p.A Milan Italy.
  • Department of Biomedical Sciences University of Padova Padua Italy.
  • Unit of Physiotherapy, Department of Health and Rehabilitation, Institute of Neuroscience and Physiology, Sahlgrenska Academy University of Gothenburg Gothenburg Sweden.
  • Sahlgrenska Sports Medicine Center, Sahlgrenska Academy Gothenburg Sweden.
  • Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg Gothenburg Sweden.
  • Department of Orthopedic Surgery, Balgrist University Hospital University of Zürich Zurich Switzerland.

ORCID-Profile: Silvia Gandini, Richard Willy, Eric Hamrin Senorski

Zitieren

APA

Gandini, S., Lindskog, J., Yu, Y., Öttl, F., Willy, R., Adami, P. E., Della Villa, F., Di Paolo, S., Bonsanto, F., Mazzoni, S., Feletti, F., Samuelsson, K., & Hamrin Senorski, E. (2026). Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus‐based bone stress injury. Journal of Experimental Orthopaedics, 13(4). https://doi.org/10.1002/jeo2.70939

Vancouver

Gandini S, Lindskog J, Yu Y, Öttl F, Willy R, Adami PE, et al. Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus‐based bone stress injury. Journal of Experimental Orthopaedics. 2026;13(4). doi:10.1002/jeo2.70939

BibTeX
@article{Gandini2026,
  author = {Gandini, Silvia and Lindskog, Jakob and Yu, Yinan and Öttl, Felix and Willy, Richard and Adami, Paolo Emilio and Della Villa, Francesco and Di Paolo, Stefano and Bonsanto, Fabio and Mazzoni, Stefano and Feletti, Francesco and Samuelsson, Kristian and Hamrin Senorski, Eric},
  title = {Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus‐based bone stress injury},
  journal = {Journal of Experimental Orthopaedics},
  year = {2026},
  volume = {13},
  number = {4},
  doi = {10.1002/jeo2.70939}
}
Kostenlos starten →