EUVIMEDEUVIMED entdecken
Delphi-StudieOpen Access · CC BY-NC 4.0

Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study

Yinan Yu iD, Luke V Tollefson, Robert F LaPrade iD, Jorge Chahla, Felix C Oettl iD, Kristian Samuelsson iD, Robert Feldt

Arthroscopy, Sports Medicine, and Rehabilitation · 09.11.2026

DOI: 10.1002/ars2.70086 ↗

Kurz gefasst

Delphi-Studie in der Zeitschrift Arthroscopy, Sports Medicine, and Rehabilitation, online erschienen am 09.11.2026. Verfasst von Yinan Yu und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering Chalmers University of Technology Gothenburg Sweden. Die Arbeit nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

To evaluate how Large Language Models (LLMs) align with a concurrently performed human medical expert‐modified Delphi study on the posterolateral corner and to qualitatively analyze LLM behavior.

Methods

Four commercially available and current LLMs (Gemini‐2.5‐Pro. GPT‐4o, o4‐mini, and o3) were tasked with rating posterolateral corner consensus statements on a 5‐point Likert scale and estimating the level of expert consensus for each statement, mirroring a human expert Delphi process. LLM performance was evaluated based on alignment with expert majority opinion, accuracy in predicting expert consensus, and self‐consistency across multiple LLM invocations. Deep research to improve LLM performance was also explored.

Results

LLM‐generated ratings showed moderate correlation with average expert ratings, with some models exhibiting tendencies toward overly positive ratings. Consensus derived from LLM self‐consistency did not reliably approximate expert consensus. LLM estimates of consensus were informative, with some models showing improved alignment with actual expert consensus. Qualitative analysis revealed distinct behavioral patterns across different LLMs regarding justification generation, certainty expression, and handling of conflicting evidence. Preliminary findings suggest that deep research capabilities can enhance rating accuracy as well as consensus prediction. Evidence of hallucination in LLM reasoning is observed and requires further verification.

Conclusions

Although LLMs show potential in reflecting expert medical knowledge and estimating consensus, their direct ratings and self‐consistency may not perfectly align with expert panels. Different LLMs display unique strengths and weaknesses, suggesting that model selection or a combination can be beneficial for applications like modified Delphi studies. Improved consensus estimation was seen by providing the models access to web search, enabling them to find recent, relevant research articles. Level of Evidence Level V, expert opinion.

Autor:innen und Institutionen

  • Department of Computer Science and Engineering Chalmers University of Technology Gothenburg Sweden.
  • Department of Computer Science and Engineering University of Gothenburg Gothenburg Sweden.
  • Center for Digital Health Sahlgrenska University Hospital Gothenburg Sweden.
  • Twin Cities Orthopedics Edina Minnesota U.S.A.
  • Rush University Medical College Chicago Illinois U.S.A.
  • Balgrist University Hospital University of Zurich Zurich Switzerland.

ORCID-Profile: Yinan Yu, Robert F LaPrade, Felix C Oettl, Kristian Samuelsson

Zitieren

APA

Yu, Y., Tollefson, L. V., LaPrade, R. F., Chahla, J., Oettl, F. C., Samuelsson, K., & Feldt, R. (2026). Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study. Arthroscopy, Sports Medicine, and Rehabilitation. https://doi.org/10.1002/ars2.70086

Vancouver

Yu Y, Tollefson LV, LaPrade RF, Chahla J, Oettl FC, Samuelsson K, et al. Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study. Arthroscopy, Sports Medicine, and Rehabilitation. 2026. doi:10.1002/ars2.70086

BibTeX
@article{Yu2026,
  author = {Yu, Yinan and Tollefson, Luke V and LaPrade, Robert F and Chahla, Jorge and Oettl, Felix C and Samuelsson, Kristian and Feldt, Robert},
  title = {Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study},
  journal = {Arthroscopy, Sports Medicine, and Rehabilitation},
  year = {2026},
  doi = {10.1002/ars2.70086}
}
Kostenlos starten →