EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Automating methodological quality assessment in orthopedic systematic reviews using large language models

Yu-Jui Huang iD, Kai-Cheng Chang, Ying-Chen Kuo, Cheng-Chen Tai, Cheng-Pang Yang

Journal of Orthopaedic Surgery · 2026-05 · 34 (2)

DOI: 10.1177/10225536261459518 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Orthopaedic Surgery, online erschienen am 07.06.2026. Verfasst von Yu-Jui Huang und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopedic Surgery, Linkou Chang Gung Memorial Hospital, Taoyuan City, Taiwan. Hauptthemen laut MEDLINE: Large Language Models, Systematic Reviews as Topic, Orthopedics. Die Arbeit nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Systematic reviews represent the foundation of evidence-based orthopedic practice, yet their methodological rigor relies heavily on accurate and consistent methodological quality assessment. This step remains time-consuming, labor-intensive, and prone to subjectivity. Recent advances in large language models (LLMs) suggest potential for automating parts of evidence synthesis.

Purpose

This study examined whether LLMs can perform AMSTAR-1–based methodological quality assessment evaluations in orthopedic systematic reviews with accuracy comparable to human experts.

Methods

Ten sports medicine knee reviews were analyzed using three LLMs—GPT-4o, GPT-5, and GPT Consensus—and their binary responses were compared against expert AMSTAR-1 ratings from a published umbrella review (110 decisions). An external validation set of four reviews published between 2022 and 2025 was included to assess generalizability and safeguard against information leakage.

Results

Agreement with human reviewers reached 87% for GPT-4o, 89% for GPT-5, and 90% for GPT Consensus; all models achieved 84% agreement in the validation set. Concordance was strongest for structured, explicitly reported domains such as a priori design, literature search, and study characteristics, and lowest for judgment-based items including grey literature inclusion, publication bias, and conflict of interest.

Conclusions

LLMs cannot yet replace human reviewers, they can serve as reliable adjunct tools to enhance efficiency, transparency, and reproducibility in systematic review workflows within orthopedic research.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsSystematic Reviews as TopicOrthopedicsHumans

Stichwörter der Autor:innen

artificial intelligenceevidence-basedlarge language modelsports

Autor:innen und Institutionen

  • Department of Orthopedic Surgery, Linkou Chang Gung Memorial Hospital, Taoyuan City, Taiwan.
  • Department of Pharmacy, Linkou Chang Gung Memorial Hospital, Taoyuan City, Taiwan.
  • Department of Physical Medicine and Rehabilitation, Shin Kong Wu Ho-Su Memorial Hospital, Taipei, Taiwan.
  • Medical Library, Department of Medical Education, Linkou Chang Gung Memorial Hospital, Taoyuan, Taiwan.

ORCID-Profile: Yu-Jui Huang

Zitieren

APA

Huang, Y. J., Chang, K. C., Kuo, Y. C., Tai, C. C., & Yang, C. P. (2026). Automating methodological quality assessment in orthopedic systematic reviews using large language models. Journal of Orthopaedic Surgery, 34(2). https://doi.org/10.1177/10225536261459518

Vancouver

Huang YJ, Chang KC, Kuo YC, Tai CC, Yang CP. Automating methodological quality assessment in orthopedic systematic reviews using large language models. Journal of Orthopaedic Surgery. 2026;34(2). doi:10.1177/10225536261459518

BibTeX
@article{Huang2026,
  author = {Huang, Yu-Jui and Chang, Kai-Cheng and Kuo, Ying-Chen and Tai, Cheng-Chen and Yang, Cheng-Pang},
  title = {Automating methodological quality assessment in orthopedic systematic reviews using large language models},
  journal = {Journal of Orthopaedic Surgery},
  year = {2026},
  volume = {34},
  number = {2},
  doi = {10.1177/10225536261459518}
}
Kostenlos starten →