EUVIMEDObjevte EUVIMED
EvaluationsstudieOpen access

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study

Arya Rao iD, Michael Pang iD, John Kim iD, Meghana Kamineni iD, Winston Lie iD, Anoop K Prasad iD, Adam Landman iD, Keith Dreyer iD, Marc D Succi iD

Journal of Medical Internet Research · 2023-08-22 · 25 · e48659

DOI: 10.2196/48659 ↗

At a glance

Evaluationsstudie in Journal of Medical Internet Research, published online on 2023-08-22. Written by Arya Rao and 8 co-authors. Institution: Medically Engineered Solutions in Healthcare Incubator, Innovation in Operations Research Center (MESH IO), Massachusetts General Hospital, Boston, M…. Major MEDLINE topics: Artificial Intelligence. Cited 364 times so far (Crossref); 29 references. The full text is freely available in PubMed Central. Funded by: NIGMS NIH HHS.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Large language model (LLM)–based artificial intelligence chatbots direct the power of large training data sets toward successive, related tasks as opposed to single-ask tasks, for which artificial intelligence already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as artificial physicians, has not yet been evaluated.

Objective

This study aimed to evaluate ChatGPT’s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes.

Methods

We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared its accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. Accuracy was measured by the proportion of correct responses to the questions posed within the clinical vignettes tested, as calculated by human scorers. We further conducted linear regression to assess the contributing factors toward ChatGPT’s performance on clinical tasks.

Results

ChatGPT achieved an overall accuracy of 71.7% (95% CI 69.3%-74.1%) across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI 67.8%-86.1%) and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI 54.2%-66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=–15.8%; P

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceHumansClinical Decision-MakingOrganizationsWorkflowUser-Centered Design

Author keywords

AIChatGPTGPTGenerative Pre-trained TransformerLLMsaccuracyartificial intelligencechatbotclinical decision supportclinical vignettesdecision-makingdevelopmentlarge language modelsusabilityutility

Authors and institutions

  • Medically Engineered Solutions in Healthcare Incubator, Innovation in Operations Research Center (MESH IO), Massachusetts General Hospital, Boston, MA, United States.
  • Harvard Medical School, Boston, MA, United States.
  • Department of Radiology, Massachusetts General Hospital, Boston, MA, United States.
  • Department of Radiology, Brigham and Women's Hospital, Boston, MA, United States.
  • Data Science Office, Mass General Brigham, Boston, MA, United States.
  • Mass General Brigham Innovation, Mass General Brigham, Boston, MA, United States.

ORCID profiles: Arya Rao, Michael Pang, John Kim, Meghana Kamineni, Winston Lie, Anoop K Prasad, Adam Landman, Keith Dreyer, Marc D Succi

Funding

  • NIGMS NIH HHS

Cite

APA

Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023). Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. Journal of Medical Internet Research, 25, e48659. https://doi.org/10.2196/48659

Vancouver

Rao A, Pang M, Kim J, Kamineni M, Lie W, Prasad AK, et al. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. Journal of Medical Internet Research. 2023;25:e48659. doi:10.2196/48659

BibTeX
@article{Rao2023,
  author = {Rao, Arya and Pang, Michael and Kim, John and Kamineni, Meghana and Lie, Winston and Prasad, Anoop K and Landman, Adam and Dreyer, Keith and Succi, Marc D},
  title = {Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study},
  journal = {Journal of Medical Internet Research},
  year = {2023},
  volume = {25},
  pages = {e48659},
  doi = {10.2196/48659}
}
Start for free →