EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas

Itamar Ben Shitrit iD, Daphna Idan, Mark Volevich, Hadar Sharabi Goldenberg, Dolev Vaknin, Or Degany iD, Nitzan Abelson, Yair Binyamin, Raouf Nassar, Majd Nassar, Aviya Kedmi, Alexander Zlotnik und 1 weitere

PLOS Digital Health · 12.03.2026 · 5 (3) · e0001278

DOI: 10.1371/journal.pdig.0001278 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 12.03.2026. Verfasst von Itamar Ben Shitrit und 12 weiteren Autor:innen. Beteiligte Einrichtung: Ben-Gurion Faculty of Health Sciences, Beer-Sheva, Israel. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Current evaluations of large language models (LLMs) in healthcare have largely emphasized theoretical benchmarks and clinician oversight, with limited exploration of real-world physician-AI interaction. In this two-stage prospective study, we assessed physician satisfaction with LLM-generated responses to real clinical queries. This study did not evaluate clinical accuracy, patient outcomes, or patient safety. In the first unblinded stage, physicians used three models - a general-purpose model (GPT-4o), a reasoning-focused model (GPT-o1), and a healthcare-specific model (OpenEvidence) - to address 25 clinical dilemmas - and rated the quality of the responses. In the second blinded stage, the same physicians evaluated responses generated either by an LLM or by a human alone, without knowledge of the source. Across 100 real-world medical responses, median physician scores on a 5-point Likert scale were comparable between unblinded and blinded evaluations (p = 0.90). Satisfaction was not associated with physicians’ resistance to change, nor did it correlate with the accuracy or relevance of cited literature. These findings suggest that physicians did not favor information generated by LLMs over externally provided responses, and that clinician satisfaction alone may not serve as a reliable proxy for validating decision support tools.

Autor:innen und Institutionen

  • Ben-Gurion Faculty of Health Sciences, Beer-Sheva, Israel.
  • Clinical Research Center, Soroka University Medical Center, Faculty of Health Sciences, Ben-Gurion University of the Negev, Beer-Sheva, Israel.
  • Technion - Israel Institute of Technology, Haifa, Israel.
  • Faculty of Medicine, Tel-Aviv University, Tel Aviv, Israel.
  • The Cheryl and Chaim Saban Children's Hospital, Soroka University Medical Center, Beer-Sheva, Israel.
  • Department of Anesthesia, Soroka University Medical Center and the Faculty of Health Sciences, Ben-Gurion University of the Negev, Beer-Sheva, Israel.

ORCID-Profile: Itamar Ben Shitrit, Or Degany

Zitieren

APA

Ben Shitrit, I., Idan, D., Volevich, M., Goldenberg, H. S., Vaknin, D., Degany, O., Abelson, N., Binyamin, Y., Nassar, R., Nassar, M., Kedmi, A., Zlotnik, A., & Einav, S. (2026). Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas. PLOS Digital Health, 5(3), e0001278. https://doi.org/10.1371/journal.pdig.0001278

Vancouver

Ben Shitrit I, Idan D, Volevich M, Goldenberg HS, Vaknin D, Degany O, et al. Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas. PLOS Digital Health. 2026;5(3):e0001278. doi:10.1371/journal.pdig.0001278

BibTeX
@article{BenShitrit2026,
  author = {Ben Shitrit, Itamar and Idan, Daphna and Volevich, Mark and Goldenberg, Hadar Sharabi and Vaknin, Dolev and Degany, Or and Abelson, Nitzan and Binyamin, Yair and Nassar, Raouf and Nassar, Majd and Kedmi, Aviya and Zlotnik, Alexander and Einav, Sharon},
  title = {Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas},
  journal = {PLOS Digital Health},
  year = {2026},
  volume = {5},
  number = {3},
  pages = {e0001278},
  doi = {10.1371/journal.pdig.0001278}
}
Kostenlos starten →