Guideline‐augmented prompting improves comparative preference and response consistency of large language model outputs for orthopaedic anaesthesia questions: A controlled prompting study
Guideline-augmentiertes Prompting verbessert Vergleichspräferenz und Antwortkonsistenz von Large Language Models bei orthopädischen Anästhesiefragen: kontrollierte Prompting-Studie
Journal of Experimental Orthopaedics · 2026-10 · 13 (4)
DOI: 10.1002/jeo2.70922 ↗Auf einen Blick
Bei 34 orthopädischen Anästhesiefragen wurden LLMs gegenüber Expert:innen bevorzugt. Guideline-augmentiertes Prompting verbesserte die Leistung, besonders bei GPT-4o.
Fragestellung
Verbessern successive LLM-Generationen und guideline-augmentiertes Prompting die klinische Übereinstimmung, Vergleichsleistung und Antwortkonsistenz bei orthopädischen Anästhesiefragen?
Methode
Kontrollierte Prompting-Studie mit 34 Fragen in 7 Subdomänen; GPT-3.5-turbo, GPT-4o und GPT-5.2 sowie 3 Expert:innen; blinded pairwise comparison durch GPT-5.2 als Judge.
Ergebnisse
Ohne Guideline-augmentiertes Prompting lagen die Win Rates bei 67,6 % (GPT-3.5-turbo), 79,4 % (GPT-4o) und 95,1 % (GPT-5.2). Mit Guideline-Augmentation stieg GPT-4o auf 91,2 % (+11,8 Prozentpunkte), GPT-5.2 auf 97,1 %.
Schlussfolgerung der Autor:innen
Successive LLM-Generationen zeigten eine schrittweise bessere Leistung beim orthopädischen Anästhesie-Reasoning. Guideline-augmentiertes Prompting verbesserte die Antwortqualität, besonders bei mittleren Modellen.
Kernaussagen
- LLMs wurden in Paarvergleichen häufiger bevorzugt als Expert:innen
- Guideline-Augmentation verbesserte die Leistung, vor allem bei GPT-4o
- Die Antwortkonsistenz variierte zwischen den Modellen
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Leitlinie in der Zeitschrift Journal of Experimental Orthopaedics, online erschienen am 03.10.2026. Verfasst von Anita Széll und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Anesthesiology and Intensive Care Medicine, Institute of Clinical Sciences Sahlgrenska Academy Gothenburg Sweden. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Purpose
Large language models (LLMs) are increasingly used in clinical contexts; however, performance in complex perioperative decision‐making remains uncertain. Orthopaedic anaesthesia presents a demanding test case due to comorbidity burden and guideline‐dependent management. Whether successive LLM generations and guideline‐augmented prompting improve clinical alignment, comparative performance and response consistency was evaluated in this study.
Methods
In this controlled prompting study, 34 orthopaedic anaesthesia questions spanning seven clinical subdomains were independently answered by three LLMs (GPT‐3.5‐turbo, GPT‐4o and GPT‐5.2) and three expert anesthesiologists. GPT‐4o and GPT‐5.2 each generated responses with and without access to relevant clinical practice guidelines. All responses were anonymized and evaluated in blinded pairwise comparisons by an independent guideline‐informed LLM‐as‐a‐judge (GPT‐5.2) using the same guideline material as the reference standard. The judge recorded preference, response consistency and confidence. Intra‐model consistency was assessed from repeated independent generations of each question.
Results
All LLMs were preferred over human experts in pairwise comparisons without guideline augmentation, with win rates of 67.6% (GPT‐3.5‐turbo), 79.4% (GPT‐4o) and 95.1% (GPT‐5.2). Performance was improved by guideline augmentation, most notably for GPT‐4o (91.2%, +11.8 percentage points), while GPT‐5.2 approached ceiling performance (97.1%). Response consistency varied between models. GPT‐4o showed the highest baseline consistency (80.4%), whereas GPT‐5.2 demonstrated no fully contradictory outputs but showed greater partial variability. Guideline augmentation numerically improved GPT‐5.2 consistency (66.7% to 80.4%) and reduced inter‐model differences. Directed qualitative analysis suggested that reviewer preference for later GPT generations was associated with greater completeness, explicit clinical reasoning and guideline‐oriented responses.
Conclusion
Successive LLM generations demonstrated progressively improved performance in orthopaedic anaesthesia reasoning. Guideline‐augmented prompting enhanced response quality, particularly for intermediate models. Guideline‐informed LLM‐as‐a‐judge evaluation appears promising for comparative assessment but requires further validation. Level of Evidence NA.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Anesthesiology and Intensive Care Medicine, Institute of Clinical Sciences Sahlgrenska Academy Gothenburg Sweden.
- Department of Anesthesiology and Intensive Care Medicine Sahlgrenska University Hospital Mölndal Mölndal Sweden.
- Department of Computer Science and Engineering Chalmers University of Technology Gothenburg Sweden.
- Department of Orthopaedics, Institute of Clinical Science Sahlgrenska Academy Gothenburg Sweden.
- Department of Anesthesiology and Intensive Care Medicine Sahlgrenska University Hospital Gothenburg Sweden.
- Clinica Ortopedica e Traumatologica II IRCCS Istituto Ortopedico Rizzoli Bologna Italy.
ORCID-Profile: Anita Széll
Zitieren
APA
Széll, A., Yu, Y., Oeding, J. F., Oettl, F. C., Piasecki, A., Dalla, K., Siöland, T., Hård Af Segerstad, M., Olsen, F., Larsson, P., Zaffagnini, S., Samuelsson, K., & Hessulf, F. (2026). Guideline‐augmented prompting improves comparative preference and response consistency of large language model outputs for orthopaedic anaesthesia questions: A controlled prompting study. Journal of Experimental Orthopaedics, 13(4). https://doi.org/10.1002/jeo2.70922
Vancouver
Széll A, Yu Y, Oeding JF, Oettl FC, Piasecki A, Dalla K, et al. Guideline‐augmented prompting improves comparative preference and response consistency of large language model outputs for orthopaedic anaesthesia questions: A controlled prompting study. Journal of Experimental Orthopaedics. 2026;13(4). doi:10.1002/jeo2.70922
BibTeX
@article{Szll2026,
author = {Széll, Anita and Yu, Yinan and Oeding, Jacob F and Oettl, Felix C and Piasecki, Adam and Dalla, Keti and Siöland, Tobias and Hård Af Segerstad, Mathias and Olsen, Fredrik and Larsson, Peter and Zaffagnini, Stefano and Samuelsson, Kristian and Hessulf, Fredrik},
title = {Guideline‐augmented prompting improves comparative preference and response consistency of large language model outputs for orthopaedic anaesthesia questions: A controlled prompting study},
journal = {Journal of Experimental Orthopaedics},
year = {2026},
volume = {13},
number = {4},
doi = {10.1002/jeo2.70922}
}
EUVIMED