Currently Available Large Language Models Do Not Provide Musculoskeletal Treatment Recommendations That Are Concordant With Evidence‐Based Clinical Practice Guidelines
Arthroscopy · 2025-02 · 41 (2) · 263
DOI: 10.1016/j.arthro.2024.07.040 ↗Kurz gefasst
Leitlinie in der Zeitschrift Arthroscopy, online erschienen am 20.08.2024. Verfasst von Benedict U Nwachukwu und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A. Hauptthemen laut MEDLINE: Practice Guidelines as Topic, Anterior Cruciate Ligament Injuries, Evidence-Based Medicine, Rotator Cuff Injuries. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 36 Literaturangaben. Gefördert von: Arthrex, Histogenics.
Abstract (Originalsprache)
Purpose
To determine whether several leading, commercially available large language models (LLMs) provide treatment recommendations concordant with evidence‐based clinical practice guidelines (CPGs) developed by the American Academy of Orthopaedic Surgeons (AAOS).
Methods
All CPGs concerning the management of rotator cuff tears (n = 33) and anterior cruciate ligament injuries (n = 15) were extracted from the AAOS. Treatment recommendations from Chat‐Generative Pretrained Transformer version 4 (ChatGPT‐4), Gemini, Mistral‐7B, and Claude‐3 were graded by 2 blinded physicians as being concordant, discordant, or indeterminate (i.e., neutral response without definitive recommendation) with respect to AAOS CPGs. The overall concordance between LLM and AAOS recommendations was quantified, and the comparative overall concordance of recommendations among the 4 LLMs was evaluated through the Fisher exact test.
Results
Overall, 135 responses (70.3%) were concordant, 43 (22.4%) were indeterminate, and 14 (7.3%) were discordant. Inter‐rater reliability for concordance classification was excellent (κ = 0.92). Concordance with AAOS CPGs was most frequently observed with ChatGPT‐4 (n = 38, 79.2%) and least frequently observed with Mistral‐7B (n = 28, 58.3%). Indeterminate recommendations were most frequently observed with Mistral‐7B (n = 17, 35.4%) and least frequently observed with Claude‐3 (n = 8, 6.7%). Discordant recommendations were most frequently observed with Gemini (n = 6, 12.5%) and least frequently observed with ChatGPT‐4 (n = 1, 2.1%). Overall, no statistically significant difference in concordant recommendations was observed across LLMs ( P = .12). Of all recommendations, only 20 (10.4%) were transparent and provided references with full bibliographic details or links to specific peer‐reviewed content to support recommendations.
Conclusions
Among leading commercially available LLMs, more than 1‐in‐4 recommendations concerning the evaluation and management of rotator cuff and anterior cruciate ligament injuries do not reflect current evidence‐based CPGs. Although ChatGPT‐4 showed the highest performance, clinically significant rates of recommendations without concordance or supporting evidence were observed. Only 10% of responses by LLMs were transparent, precluding users from fully interpreting the sources from which recommendations were provided. Clinical Relevance Although leading LLMs generally provide recommendations concordant with CPGs, a substantial error rate exists, and the proportion of recommendations that do not align with these CPGs suggests that LLMs are not trustworthy clinical support tools at this time. Each off‐the‐shelf, closed‐source LLM has strengths and weaknesses. Future research should evaluate and compare multiple LLMs to avoid bias associated with narrow evaluation of few models as observed in the current literature.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A.
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A.. Electronic address: Kylekunze7@gmail.com.
Förderung
- Arthrex
- Histogenics
Zitieren
APA
Nwachukwu, B. U., Varady, N. H., Allen, A. A., Dines, J. S., Altchek, D. W., Williams, R. J., & Kunze, K. N. (2025). Currently Available Large Language Models Do Not Provide Musculoskeletal Treatment Recommendations That Are Concordant With Evidence‐Based Clinical Practice Guidelines. Arthroscopy, 41(2), 263. https://doi.org/10.1016/j.arthro.2024.07.040
Vancouver
Nwachukwu BU, Varady NH, Allen AA, Dines JS, Altchek DW, Williams RJ, et al. Currently Available Large Language Models Do Not Provide Musculoskeletal Treatment Recommendations That Are Concordant With Evidence‐Based Clinical Practice Guidelines. Arthroscopy. 2025;41(2):263. doi:10.1016/j.arthro.2024.07.040
BibTeX
@article{Nwachukwu2025,
author = {Nwachukwu, Benedict U and Varady, Nathan H and Allen, Answorth A and Dines, Joshua S and Altchek, David W and Williams, Riley J and Kunze, Kyle N},
title = {Currently Available Large Language Models Do Not Provide Musculoskeletal Treatment Recommendations That Are Concordant With Evidence‐Based Clinical Practice Guidelines},
journal = {Arthroscopy},
year = {2025},
volume = {41},
number = {2},
pages = {263},
doi = {10.1016/j.arthro.2024.07.040}
}
EUVIMED