Evaluating the Usability, Technical Performance, and Accuracy of Artificial Intelligence Scribes for Primary Care: Competitive Analysis
JMIR Human Factors · 2025-07-23 · 12 · e71434-e71434
DOI: 10.2196/71434 ↗At a glance
Journal article in JMIR Human Factors, published online on 2025-07-23. Written by Emily Ha and 6 co-authors. Institution: Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada. Major MEDLINE topics: Artificial Intelligence, Primary Health Care, Documentation, Electronic Health Records. Cited 51 times so far (Crossref); 48 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Primary care providers (PCPs) face significant burnout due to increasing administrative and documentation demands, contributing to job dissatisfaction and impacting care quality. Artificial intelligence (AI) scribes have emerged as potential solutions to reduce administrative burden by automating clinical documentation of patient encounters. Although AI scribes are gaining popularity in primary care, there is limited information on their usability, effectiveness, and accuracy.
Objective
This study aimed to develop and apply an evaluation framework to systematically assess the usability, technical performance, and accuracy of various AI scribes used in primary care settings across Canada and the United States.
Methods
We conducted a systematic comparison of a suite of AI scribes using competitive analysis methods. An evaluation framework was developed using expert usability approaches and human factors engineering principles and comprises 3 domains: usability, effectiveness and technical performance, and accuracy and quality. Audio files from 4 standardized patient encounters were used to generate transcripts and SOAP (Subjective, Objective, Assessment, and Plan)–format medical notes from each AI scribe. A verbatim transcript, detailed case notes, and physician-written medical notes for each audio file served as a benchmark for comparison against the AI-generated outputs. Applicable items were rated on a 3-point Likert scale (1=poor, 2=good, 3=excellent). Additional insights were gathered from clinical experts, vendor questionnaires, and public resources to support usability, effectiveness, and quality findings.
Results
In total, 6 AI scribes were evaluated, with notable performance differences. Most AI scribes could be accessed via various platforms (n=4) and launched within common electronic medical records, though data exchange capabilities were limited. Nearly all AI scribes generated SOAP-format notes in approximately 1 minute for a 15-minute standardized encounter (n=5), though documentation time increased with encounter length and topic complexity. While all AI scribes produced good to excellent quality medical notes, none were consistently error-free. Common errors included deletion, omission, and SOAP structure errors. Factors such as extraneous conversations and multiple speakers impacted the accuracy of both the transcript and medical note, with some AI scribes producing excellent notes despite minor transcript issues and vice versa. Limitations in usability, technical performance, and accuracy suggest areas for improvement to fully realize AI scribes’ potential in reducing administrative burden for PCPs.
Conclusions
This study offers one of the first systematic evaluations of the usability, effectiveness, and accuracy of a suite of AI scribes currently used in primary care, providing benchmark data for further research, policy, and practice. While AI scribes show promise in reducing documentation burdens, improvements and ongoing evaluations are essential to ensure safe and effective use. Future studies should assess AI scribe performance in real-world settings across diverse populations to support equitable and reliable applications.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada.
- Women's College Hospital Institute for Health System Solutions and Virtual Care, Women's College Hospital, 76 Grenville Street, 6th Floor, Toronto, ON, M5S 1B2, Canada, 1 4163236400.
- Institute for Better Health, Trillium Health Partners, Mississauga, ON, Canada.
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada.
- Institute for Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada.
- Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
ORCID profiles: Emily Ha, Isabelle Choon-Kon-Yune, LaShawn Murray, Siying Luan, Enid Montague, Onil Bhattacharyya, Payal Agarwal
Cite
APA
Ha, E., Choon-Kon-Yune, I., Murray, L., Luan, S., Montague, E., Bhattacharyya, O., & Agarwal, P. (2025). Evaluating the Usability, Technical Performance, and Accuracy of Artificial Intelligence Scribes for Primary Care: Competitive Analysis. JMIR Human Factors, 12, e71434-e71434. https://doi.org/10.2196/71434
Vancouver
Ha E, Choon-Kon-Yune I, Murray L, Luan S, Montague E, Bhattacharyya O, et al. Evaluating the Usability, Technical Performance, and Accuracy of Artificial Intelligence Scribes for Primary Care: Competitive Analysis. JMIR Human Factors. 2025;12:e71434-e71434. doi:10.2196/71434
BibTeX
@article{Ha2025,
author = {Ha, Emily and Choon-Kon-Yune, Isabelle and Murray, LaShawn and Luan, Siying and Montague, Enid and Bhattacharyya, Onil and Agarwal, Payal},
title = {Evaluating the Usability, Technical Performance, and Accuracy of Artificial Intelligence Scribes for Primary Care: Competitive Analysis},
journal = {JMIR Human Factors},
year = {2025},
volume = {12},
pages = {e71434--e71434},
doi = {10.2196/71434}
}
EUVIMED