EUVIMEDEUVIMED entdecken
Randomisierte kontrollierte StudieOpen Access

Large Language Model Influence on Diagnostic Reasoning

Ethan Goh, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Joséphine A Cool, Zahir Kanjee, Andrew S Parsons, Neera Ahuja, Eric Horvitz, Daniel Yang und 4 weitere

JAMA Network Open · 28.10.2024 · 7 (10) · e2440969

DOI: 10.1001/jamanetworkopen.2024.40969 ↗

Kurz gefasst

Randomisierte kontrollierte Studie in der Zeitschrift JAMA Network Open, online erschienen am 28.10.2024. Verfasst von Ethan Goh und 15 weiteren Autor:innen. Beteiligte Einrichtung: Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California. Hauptthemen laut MEDLINE: Clinical Reasoning, Clinical Competence. Die Arbeit wurde bisher 806-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIAID NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Importance Large language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such tools improves physician diagnostic reasoning.

Objective

To assess the effect of an LLM on physicians’ diagnostic reasoning compared with conventional resources. Design, Setting, and Participants A single-blind randomized clinical trial was conducted from November 29 to December 29, 2023. Using remote video conferencing and in-person participation across multiple academic medical institutions, physicians with training in family medicine, internal medicine, or emergency medicine were recruited.

Intervention

Participants were randomized to either access the LLM in addition to conventional diagnostic resources or conventional resources only, stratified by career stage. Participants were allocated 60 minutes to review up to 6 clinical vignettes. Main Outcomes and Measures The primary outcome was performance on a standardized rubric of diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps, validated and graded via blinded expert consensus. Secondary outcomes included time spent per case (in seconds) and final diagnosis accuracy. All analyses followed the intention-to-treat principle. A secondary exploratory analysis evaluated the standalone performance of the LLM by comparing the primary outcomes between the LLM alone group and the conventional resource group.

Results

Fifty physicians (26 attendings, 24 residents; median years in practice, 3 [IQR, 2-8]) participated virtually as well as at 1 in-person site. The median diagnostic reasoning score per case was 76% (IQR, 66%-87%) for the LLM group and 74% (IQR, 63%-84%) for the conventional resources-only group, with an adjusted difference of 2 percentage points (95% CI, −4 to 8 percentage points; P = .60). The median time spent per case for the LLM group was 519 (IQR, 371-668) seconds, compared with 565 (IQR, 456-788) seconds for the conventional resources group, with a time difference of −82 (95% CI, −195 to 31; P = .20) seconds. The LLM alone scored 16 percentage points (95% CI, 2-30 percentage points; P = .03) higher than the conventional resources group. Conclusions and Relevance In this trial, the availability of an LLM to physicians as a diagnostic aid did not significantly improve clinical reasoning compared with conventional resources. The LLM alone demonstrated higher performance than both physician groups, indicating the need for technology and workforce development to realize the potential of physician-artificial intelligence collaboration in clinical practice. Trial Registration ClinicalTrials.gov Identifier: NCT06157944

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Clinical ReasoningClinical CompetenceHumansFemaleMaleSingle-Blind MethodAdultLanguagePhysicians

Autor:innen und Institutionen

  • Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California.
  • Stanford Clinical Excellence Research Center, Stanford University, Stanford, California.
  • Center for Innovation to Implementation, VA Palo Alto Health Care System, Palo Alto, California.
  • Department of Hospital Medicine, Stanford University School of Medicine, Stanford, California.
  • Quantitative Sciences Unit, Stanford University School of Medicine, Stanford, California.
  • Department of Hospital Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts.

Förderung

  • NIAID NIH HHS

Zitieren

APA

Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Rodman, A., & Chen, J. H. (2024). Large Language Model Influence on Diagnostic Reasoning. JAMA Network Open, 7(10), e2440969. https://doi.org/10.1001/jamanetworkopen.2024.40969

Vancouver

Goh E, Gallo R, Hom J, Strong E, Weng Y, Kerman H, et al. Large Language Model Influence on Diagnostic Reasoning. JAMA Network Open. 2024;7(10):e2440969. doi:10.1001/jamanetworkopen.2024.40969

BibTeX
@article{Goh2024,
  author = {Goh, Ethan and Gallo, Robert and Hom, Jason and Strong, Eric and Weng, Yingjie and Kerman, Hannah and Cool, Joséphine A and Kanjee, Zahir and Parsons, Andrew S and Ahuja, Neera and Horvitz, Eric and Yang, Daniel and Milstein, Arnold and Olson, Andrew P J and Rodman, Adam and Chen, Jonathan H},
  title = {Large Language Model Influence on Diagnostic Reasoning},
  journal = {JAMA Network Open},
  year = {2024},
  volume = {7},
  number = {10},
  pages = {e2440969},
  doi = {10.1001/jamanetworkopen.2024.40969}
}
Kostenlos starten →