Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study
Arthroscopy, Sports Medicine, and Rehabilitation · 09.11.2026
DOI: 10.1002/ars2.70086 ↗Kurz gefasst
Delphi-Studie in der Zeitschrift Arthroscopy, Sports Medicine, and Rehabilitation, online erschienen am 09.11.2026. Verfasst von Yinan Yu und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering Chalmers University of Technology Gothenburg Sweden. Die Arbeit nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Purpose
To evaluate how Large Language Models (LLMs) align with a concurrently performed human medical expert‐modified Delphi study on the posterolateral corner and to qualitatively analyze LLM behavior.
Methods
Four commercially available and current LLMs (Gemini‐2.5‐Pro. GPT‐4o, o4‐mini, and o3) were tasked with rating posterolateral corner consensus statements on a 5‐point Likert scale and estimating the level of expert consensus for each statement, mirroring a human expert Delphi process. LLM performance was evaluated based on alignment with expert majority opinion, accuracy in predicting expert consensus, and self‐consistency across multiple LLM invocations. Deep research to improve LLM performance was also explored.
Results
LLM‐generated ratings showed moderate correlation with average expert ratings, with some models exhibiting tendencies toward overly positive ratings. Consensus derived from LLM self‐consistency did not reliably approximate expert consensus. LLM estimates of consensus were informative, with some models showing improved alignment with actual expert consensus. Qualitative analysis revealed distinct behavioral patterns across different LLMs regarding justification generation, certainty expression, and handling of conflicting evidence. Preliminary findings suggest that deep research capabilities can enhance rating accuracy as well as consensus prediction. Evidence of hallucination in LLM reasoning is observed and requires further verification.
Conclusions
Although LLMs show potential in reflecting expert medical knowledge and estimating consensus, their direct ratings and self‐consistency may not perfectly align with expert panels. Different LLMs display unique strengths and weaknesses, suggesting that model selection or a combination can be beneficial for applications like modified Delphi studies. Improved consensus estimation was seen by providing the models access to web search, enabling them to find recent, relevant research articles. Level of Evidence Level V, expert opinion.
Autor:innen und Institutionen
- Department of Computer Science and Engineering Chalmers University of Technology Gothenburg Sweden.
- Department of Computer Science and Engineering University of Gothenburg Gothenburg Sweden.
- Center for Digital Health Sahlgrenska University Hospital Gothenburg Sweden.
- Twin Cities Orthopedics Edina Minnesota U.S.A.
- Rush University Medical College Chicago Illinois U.S.A.
- Balgrist University Hospital University of Zurich Zurich Switzerland.
ORCID-Profile: Yinan Yu, Robert F LaPrade, Felix C Oettl, Kristian Samuelsson
Zitieren
APA
Yu, Y., Tollefson, L. V., LaPrade, R. F., Chahla, J., Oettl, F. C., Samuelsson, K., & Feldt, R. (2026). Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study. Arthroscopy, Sports Medicine, and Rehabilitation. https://doi.org/10.1002/ars2.70086
Vancouver
Yu Y, Tollefson LV, LaPrade RF, Chahla J, Oettl FC, Samuelsson K, et al. Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study. Arthroscopy, Sports Medicine, and Rehabilitation. 2026. doi:10.1002/ars2.70086
BibTeX
@article{Yu2026,
author = {Yu, Yinan and Tollefson, Luke V and LaPrade, Robert F and Chahla, Jorge and Oettl, Felix C and Samuelsson, Kristian and Feldt, Robert},
title = {Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study},
journal = {Arthroscopy, Sports Medicine, and Rehabilitation},
year = {2026},
doi = {10.1002/ars2.70086}
}
EUVIMED