EUVIMEDScopri EUVIMED
Randomisierte kontrollierte StudieOpen access

Effect of Uncertainty-Aware AI Models on Pharmacists’ Reaction Time and Decision-Making in a Web-Based Mock Medication Verification Task: Randomized Controlled Trial

Corey Lester iD, Brigid Rowell iD, Yifan Zheng iD, Zoe Co iD, Vincent Marshall iD, Jin Yong Kim iD, Qiyuan Chen iD, Raed Kontar iD, X Jessie Yang iD

JMIR Medical Informatics · 2025

DOI: 10.2196/64902 ↗

At a glance

Randomisierte kontrollierte Studie in JMIR Medical Informatics, 2025. Written by Corey Lester and 8 co-authors. Institution: Department of Clinical Pharmacy, College of Pharmacy, University of Michigan, 428 Church Street, Ann Arbor, MI, 48109, United States, 1 734-647-8849. Major MEDLINE topics: Pharmacists, Artificial Intelligence, Reaction Time, Decision Making. Cited 15 times so far (Crossref); 45 references. The full text is freely available in PubMed Central. Funded by: NLM NIH HHS.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Artificial intelligence (AI)–based clinical decision support systems are increasingly used in health care. Uncertainty-aware AI presents the model’s confidence in its decision alongside its prediction, whereas black-box AI only provides a prediction. Little is known about how this type of AI affects health care providers’ work performance and reaction time.

Objective

This study aimed to determine the effects of black-box and uncertainty-aware AI advice on pharmacist decision-making and reaction time.

Methods

Recruitment emails were sent to pharmacists through professional listservs describing a web-based, crossover, randomized controlled trial. Participants were randomized to the black-box AI or uncertainty-aware AI condition in a 1:1 manner. Participants completed 100 mock verification tasks with AI help and 100 without AI help. The order of no help and AI help was randomized. Participants were exposed to correct and incorrect prescription fills, where the correct decision was to “accept” or “reject,” respectively. AI help provided correct (79%) or incorrect (21%) advice. Reaction times, participant decisions, AI advice, and AI help type were recorded for each verification. Likelihood ratio tests compared means across the three categories of AI type for each level of AI correctness.

Results

A total of 30 participants provided complete datasets. An equal number of participants were in each AI condition. Participants’ decision-making performance and reaction times differed across the 3 conditions. Accurate AI recommendations resulted in the rejection of the incorrect drug 96.1% and 91.8% of the time for uncertainty-aware AI and black-box AI respectively, compared with 81.2% without AI help. Correctly dispensed medications were accepted at rates of 99.2% with black-box help, 94.1% with uncertainty-aware AI help, and 94.6% without AI help. Uncertainty-aware AI protected against bad AI advice to approve an incorrectly filled medication compared with black-box AI (83.3% vs 76.7%). When the AI recommended rejecting a correctly filled medication, pharmacists without AI help had a higher rate of correctly accepting the medication (94.6%) compared with uncertainty-aware AI help (86.2%) and black-box AI help (81.2%). Uncertainty-aware AI resulted in shorter reaction times than black-box AI and no AI help except in the scenario where “AI rejects the correct drug.” Black-box AI did not lead to reduced reaction times compared with pharmacists acting alone.

Conclusions

Pharmacists’ performance and reaction times varied by AI type and AI accuracy. Overall, uncertainty-aware AI resulted in faster decision-making and acted as a safeguard against bad AI advice to approve a misfilled medication. Conversely, black-box AI had the longest reaction times, and user performance degraded in the presence of bad AI advice. However, uncertainty-aware AI could result in unnecessary double-checks, but it is preferred over false negative advice, where patients receive the wrong medication. These results highlight the importance of well-designed AI that addresses users’ needs, enhances performance, and avoids overreliance on AI.

Subjects

MeSH terms (MEDLINE)

PharmacistsArtificial IntelligenceReaction TimeDecision MakingHumansMaleUncertaintyFemaleAdultInternetDecision Support Systems, ClinicalCross-Over StudiesMiddle Aged

Author keywords

clinical decision support systemsAIartificial intelligenceclinical decision supportdecision-makingdiagnosisdrugdrug developmenthuman factorshuman-computer interactionpharmacistpredictionrandomized controlled trialverification

Authors and institutions

  • Department of Clinical Pharmacy, College of Pharmacy, University of Michigan, 428 Church Street, Ann Arbor, MI, 48109, United States, 1 734-647-8849.
  • Department of Learning Health Sciences, University of Michigan School of Medicine, Ann Arbor, MI, United States.
  • Department of Industrial and Operations Engineering, College of Engineering, University of Michigan, Ann Arbor, MI, United States.

ORCID profiles: Corey Lester, Brigid Rowell, Yifan Zheng, Zoe Co, Vincent Marshall, Jin Yong Kim, Qiyuan Chen, Raed Kontar, X Jessie Yang

Funding

  • NLM NIH HHS

Cite

APA

Lester, C., Rowell, B., Zheng, Y., Co, Z., Marshall, V., Kim, J. Y., Chen, Q., Kontar, R., & Yang, X. J. (2025). Effect of Uncertainty-Aware AI Models on Pharmacists’ Reaction Time and Decision-Making in a Web-Based Mock Medication Verification Task: Randomized Controlled Trial. JMIR Medical Informatics. https://doi.org/10.2196/64902

Vancouver

Lester C, Rowell B, Zheng Y, Co Z, Marshall V, Kim JY, et al. Effect of Uncertainty-Aware AI Models on Pharmacists’ Reaction Time and Decision-Making in a Web-Based Mock Medication Verification Task: Randomized Controlled Trial. JMIR Medical Informatics. 2025. doi:10.2196/64902

BibTeX
@article{Lester2025,
  author = {Lester, Corey and Rowell, Brigid and Zheng, Yifan and Co, Zoe and Marshall, Vincent and Kim, Jin Yong and Chen, Qiyuan and Kontar, Raed and Yang, X Jessie},
  title = {Effect of Uncertainty-Aware AI Models on Pharmacists’ Reaction Time and Decision-Making in a Web-Based Mock Medication Verification Task: Randomized Controlled Trial},
  journal = {JMIR Medical Informatics},
  year = {2025},
  doi = {10.2196/64902}
}
Start for free →