Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas
PLOS Digital Health · 12.03.2026 · 5 (3) · e0001278
DOI: 10.1371/journal.pdig.0001278 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 12.03.2026. Verfasst von Itamar Ben Shitrit und 12 weiteren Autor:innen. Beteiligte Einrichtung: Ben-Gurion Faculty of Health Sciences, Beer-Sheva, Israel. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Current evaluations of large language models (LLMs) in healthcare have largely emphasized theoretical benchmarks and clinician oversight, with limited exploration of real-world physician-AI interaction. In this two-stage prospective study, we assessed physician satisfaction with LLM-generated responses to real clinical queries. This study did not evaluate clinical accuracy, patient outcomes, or patient safety. In the first unblinded stage, physicians used three models - a general-purpose model (GPT-4o), a reasoning-focused model (GPT-o1), and a healthcare-specific model (OpenEvidence) - to address 25 clinical dilemmas - and rated the quality of the responses. In the second blinded stage, the same physicians evaluated responses generated either by an LLM or by a human alone, without knowledge of the source. Across 100 real-world medical responses, median physician scores on a 5-point Likert scale were comparable between unblinded and blinded evaluations (p = 0.90). Satisfaction was not associated with physicians’ resistance to change, nor did it correlate with the accuracy or relevance of cited literature. These findings suggest that physicians did not favor information generated by LLMs over externally provided responses, and that clinician satisfaction alone may not serve as a reliable proxy for validating decision support tools.
Autor:innen und Institutionen
- Ben-Gurion Faculty of Health Sciences, Beer-Sheva, Israel.
- Clinical Research Center, Soroka University Medical Center, Faculty of Health Sciences, Ben-Gurion University of the Negev, Beer-Sheva, Israel.
- Technion - Israel Institute of Technology, Haifa, Israel.
- Faculty of Medicine, Tel-Aviv University, Tel Aviv, Israel.
- The Cheryl and Chaim Saban Children's Hospital, Soroka University Medical Center, Beer-Sheva, Israel.
- Department of Anesthesia, Soroka University Medical Center and the Faculty of Health Sciences, Ben-Gurion University of the Negev, Beer-Sheva, Israel.
ORCID-Profile: Itamar Ben Shitrit, Or Degany
Zitieren
APA
Ben Shitrit, I., Idan, D., Volevich, M., Goldenberg, H. S., Vaknin, D., Degany, O., Abelson, N., Binyamin, Y., Nassar, R., Nassar, M., Kedmi, A., Zlotnik, A., & Einav, S. (2026). Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas. PLOS Digital Health, 5(3), e0001278. https://doi.org/10.1371/journal.pdig.0001278
Vancouver
Ben Shitrit I, Idan D, Volevich M, Goldenberg HS, Vaknin D, Degany O, et al. Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas. PLOS Digital Health. 2026;5(3):e0001278. doi:10.1371/journal.pdig.0001278
BibTeX
@article{BenShitrit2026,
author = {Ben Shitrit, Itamar and Idan, Daphna and Volevich, Mark and Goldenberg, Hadar Sharabi and Vaknin, Dolev and Degany, Or and Abelson, Nitzan and Binyamin, Yair and Nassar, Raouf and Nassar, Majd and Kedmi, Aviya and Zlotnik, Alexander and Einav, Sharon},
title = {Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas},
journal = {PLOS Digital Health},
year = {2026},
volume = {5},
number = {3},
pages = {e0001278},
doi = {10.1371/journal.pdig.0001278}
}
EUVIMED