Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study
JMIR Medical Informatics · 17.04.2026 · 14 · e86474-e86474
DOI: 10.2196/86474 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Informatics, online erschienen am 17.04.2026. Verfasst von Sandra L Taylor und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Public Health Sciences, School of Medicine, University of California, Davis, 4480 2nd Avenue, Suite 4152, Sacramento, CA, 95817, United…. Hauptthemen laut MEDLINE: Documentation, Electronic Health Records, Artificial Intelligence. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS.
Abstract (Originalsprache)
Background
Physicians routinely document specifics of patient encounters in clinic visit notes, a critical but potentially time-consuming task. Ambient listening artificial intelligence (AI) technology is being integrated into clinical workflows to reduce documentation burden by creating draft visit notes. While this technology is promising, it is not perfect, and the potential for patient harm needs to be understood and mitigated. We developed and piloted an efficient, standardized approach to evaluating AI-generated notes for safety concerns in ambulatory care visits.
Objective
The objective of this quality improvement project was to develop and pilot an efficient, standardized, and scalable approach to evaluating AI-generated notes for safety concerns in ambulatory care visits.
Methods
During a 2-month pilot (July to August 2024), 31 physicians across multiple specialties used an ambient listening AI scribe to assist with the creation of 7545 clinic notes. A novel survey instrument was developed to assess note quality, focusing on 4 error types: accidental inclusions, accidental omissions, hallucinations, and bias. Physicians evaluated 356 (4.7%) AI-generated notes. Where an error was present, physicians rated its severity based on its potential to cause patient harm if it was not corrected, on a 0 to 5 scale. Additionally, a vendor-reported metric on the percentage of note content edited by physicians was analyzed.
Results
Of the 356 evaluated notes, accidental omissions were the most frequent error (n=64, 18%), followed by hallucinations (n=41, 11.5%), and accidental inclusions (n=33, 9.3%). Bias was rare (n=4, 1.1%). Most (119/142, 83.8%) errors were rated as mild to moderate (severity 1‐3), with only 19 (5.3%) notes containing errors rated as posing serious or imminent risk (severity 4‐5). Editing metrics across all AI-created notes showed a median of 9.0% (IQR 2.5%-21.9%) of AI-generated words were changed, with 14.9% (143/960) of notes left entirely unedited. Physician editing practices varied widely, with average percentages of AI-generated words changed ranging from 1.9% to 69.3% (median 9.0%, IQR 2.5%-21.9%).
Conclusions
AI-generated clinical notes were generally of high quality, with 94.7% (337/356) free from significant errors. However, because a small number contained errors that carried the risk of serious harm if not corrected, careful clinician review of notes remains imperative. Prior to deploying an AI scribe, organizations should pilot the technology and include an efficient review process to understand the nature and type of errors common at their organization. This pilot provides a scalable model for other health systems seeking to implement AI scribe technology responsibly.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Public Health Sciences, School of Medicine, University of California, Davis, 4480 2nd Avenue, Suite 4152, Sacramento, CA, 95817, United States, 1 916-734-4800.
- Clinical and Translational Science Center, University of California, Davis, Sacramento, CA, United States.
- Department of Clinical Informatics, University of California, Davis, Sacramento, CA, United States.
- IT Clinical Applications, University of California, Davis, Sacramento, CA, United States.
- Blaisdell Medical Library, University of California, Davis, Sacramento, CA, United States.
- Department of Otolaryngology, Head and Neck Surgery, School of Medicine, University of California, Davis, Sacramento, CA, United States.
ORCID-Profile: Sandra L Taylor, Melissa Jost, Scott MacDonald, Yunyi Ren, Shelley Hilton, Sadie Davenport, Debbie Aizenberg, Bruce Hall, Courtney R Lyles, Jason Y Adams
Förderung
- NCATS NIH HHS
Zitieren
APA
Taylor, S. L., Jost, M., MacDonald, S., Ren, Y., Hilton, S., Davenport, S., Aizenberg, D., Hall, B., Lyles, C. R., & Adams, J. Y. (2026). Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study. JMIR Medical Informatics, 14, e86474-e86474. https://doi.org/10.2196/86474
Vancouver
Taylor SL, Jost M, MacDonald S, Ren Y, Hilton S, Davenport S, et al. Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study. JMIR Medical Informatics. 2026;14:e86474-e86474. doi:10.2196/86474
BibTeX
@article{Taylor2026,
author = {Taylor, Sandra L and Jost, Melissa and MacDonald, Scott and Ren, Yunyi and Hilton, Shelley and Davenport, Sadie and Aizenberg, Debbie and Hall, Bruce and Lyles, Courtney R and Adams, Jason Y},
title = {Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study},
journal = {JMIR Medical Informatics},
year = {2026},
volume = {14},
pages = {e86474--e86474},
doi = {10.2196/86474}
}
EUVIMED