Automating methodological quality assessment in orthopedic systematic reviews using large language models
Journal of Orthopaedic Surgery · 2026-05 · 34 (2)
DOI: 10.1177/10225536261459518 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Orthopaedic Surgery, online erschienen am 07.06.2026. Verfasst von Yu-Jui Huang und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopedic Surgery, Linkou Chang Gung Memorial Hospital, Taoyuan City, Taiwan. Hauptthemen laut MEDLINE: Large Language Models, Systematic Reviews as Topic, Orthopedics. Die Arbeit nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Background
Systematic reviews represent the foundation of evidence-based orthopedic practice, yet their methodological rigor relies heavily on accurate and consistent methodological quality assessment. This step remains time-consuming, labor-intensive, and prone to subjectivity. Recent advances in large language models (LLMs) suggest potential for automating parts of evidence synthesis.
Purpose
This study examined whether LLMs can perform AMSTAR-1–based methodological quality assessment evaluations in orthopedic systematic reviews with accuracy comparable to human experts.
Methods
Ten sports medicine knee reviews were analyzed using three LLMs—GPT-4o, GPT-5, and GPT Consensus—and their binary responses were compared against expert AMSTAR-1 ratings from a published umbrella review (110 decisions). An external validation set of four reviews published between 2022 and 2025 was included to assess generalizability and safeguard against information leakage.
Results
Agreement with human reviewers reached 87% for GPT-4o, 89% for GPT-5, and 90% for GPT Consensus; all models achieved 84% agreement in the validation set. Concordance was strongest for structured, explicitly reported domains such as a priori design, literature search, and study characteristics, and lowest for judgment-based items including grey literature inclusion, publication bias, and conflict of interest.
Conclusions
LLMs cannot yet replace human reviewers, they can serve as reliable adjunct tools to enhance efficiency, transparency, and reproducibility in systematic review workflows within orthopedic research.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Orthopedic Surgery, Linkou Chang Gung Memorial Hospital, Taoyuan City, Taiwan.
- Department of Pharmacy, Linkou Chang Gung Memorial Hospital, Taoyuan City, Taiwan.
- Department of Physical Medicine and Rehabilitation, Shin Kong Wu Ho-Su Memorial Hospital, Taipei, Taiwan.
- Medical Library, Department of Medical Education, Linkou Chang Gung Memorial Hospital, Taoyuan, Taiwan.
ORCID-Profile: Yu-Jui Huang
Zitieren
APA
Huang, Y. J., Chang, K. C., Kuo, Y. C., Tai, C. C., & Yang, C. P. (2026). Automating methodological quality assessment in orthopedic systematic reviews using large language models. Journal of Orthopaedic Surgery, 34(2). https://doi.org/10.1177/10225536261459518
Vancouver
Huang YJ, Chang KC, Kuo YC, Tai CC, Yang CP. Automating methodological quality assessment in orthopedic systematic reviews using large language models. Journal of Orthopaedic Surgery. 2026;34(2). doi:10.1177/10225536261459518
BibTeX
@article{Huang2026,
author = {Huang, Yu-Jui and Chang, Kai-Cheng and Kuo, Ying-Chen and Tai, Cheng-Chen and Yang, Cheng-Pang},
title = {Automating methodological quality assessment in orthopedic systematic reviews using large language models},
journal = {Journal of Orthopaedic Surgery},
year = {2026},
volume = {34},
number = {2},
doi = {10.1177/10225536261459518}
}
EUVIMED