EUVIMEDObjevte EUVIMED
Open access

Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study

Sandra L Taylor iD, Melissa Jost iD, Scott MacDonald iD, Yunyi Ren iD, Shelley Hilton iD, Sadie Davenport iD, Debbie Aizenberg iD, Bruce Hall iD, Courtney R Lyles iD, Jason Y Adams iD

JMIR Medical Informatics · 2026-04-17 · 14 · e86474-e86474

DOI: 10.2196/86474 ↗

At a glance

Journal article in JMIR Medical Informatics, published online on 2026-04-17. Written by Sandra L Taylor and 9 co-authors. Institution: Department of Public Health Sciences, School of Medicine, University of California, Davis, 4480 2nd Avenue, Suite 4152, Sacramento, CA, 95817, United…. Major MEDLINE topics: Documentation, Electronic Health Records, Artificial Intelligence. Cited 11 times so far (Crossref); 13 references. The full text is freely available in PubMed Central. Funded by: NCATS NIH HHS.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Physicians routinely document specifics of patient encounters in clinic visit notes, a critical but potentially time-consuming task. Ambient listening artificial intelligence (AI) technology is being integrated into clinical workflows to reduce documentation burden by creating draft visit notes. While this technology is promising, it is not perfect, and the potential for patient harm needs to be understood and mitigated. We developed and piloted an efficient, standardized approach to evaluating AI-generated notes for safety concerns in ambulatory care visits.

Objective

The objective of this quality improvement project was to develop and pilot an efficient, standardized, and scalable approach to evaluating AI-generated notes for safety concerns in ambulatory care visits.

Methods

During a 2-month pilot (July to August 2024), 31 physicians across multiple specialties used an ambient listening AI scribe to assist with the creation of 7545 clinic notes. A novel survey instrument was developed to assess note quality, focusing on 4 error types: accidental inclusions, accidental omissions, hallucinations, and bias. Physicians evaluated 356 (4.7%) AI-generated notes. Where an error was present, physicians rated its severity based on its potential to cause patient harm if it was not corrected, on a 0 to 5 scale. Additionally, a vendor-reported metric on the percentage of note content edited by physicians was analyzed.

Results

Of the 356 evaluated notes, accidental omissions were the most frequent error (n=64, 18%), followed by hallucinations (n=41, 11.5%), and accidental inclusions (n=33, 9.3%). Bias was rare (n=4, 1.1%). Most (119/142, 83.8%) errors were rated as mild to moderate (severity 1‐3), with only 19 (5.3%) notes containing errors rated as posing serious or imminent risk (severity 4‐5). Editing metrics across all AI-created notes showed a median of 9.0% (IQR 2.5%-21.9%) of AI-generated words were changed, with 14.9% (143/960) of notes left entirely unedited. Physician editing practices varied widely, with average percentages of AI-generated words changed ranging from 1.9% to 69.3% (median 9.0%, IQR 2.5%-21.9%).

Conclusions

AI-generated clinical notes were generally of high quality, with 94.7% (337/356) free from significant errors. However, because a small number contained errors that carried the risk of serious harm if not corrected, careful clinician review of notes remains imperative. Prior to deploying an AI scribe, organizations should pilot the technology and include an efficient review process to understand the nature and type of errors common at their organization. This pilot provides a scalable model for other health systems seeking to implement AI scribe technology responsibly.

Subjects

MeSH terms (MEDLINE)

DocumentationElectronic Health RecordsArtificial IntelligencePilot ProjectsHumansProspective StudiesGenerative Artificial IntelligenceQuality ImprovementSurveys and Questionnaires

Author keywords

AI scribeambient listeningartificial intelligence scribeclinical documentationgenerative AIgenerative artificial intelligencequality of care

Authors and institutions

  • Department of Public Health Sciences, School of Medicine, University of California, Davis, 4480 2nd Avenue, Suite 4152, Sacramento, CA, 95817, United States, 1 916-734-4800.
  • Clinical and Translational Science Center, University of California, Davis, Sacramento, CA, United States.
  • Department of Clinical Informatics, University of California, Davis, Sacramento, CA, United States.
  • IT Clinical Applications, University of California, Davis, Sacramento, CA, United States.
  • Blaisdell Medical Library, University of California, Davis, Sacramento, CA, United States.
  • Department of Otolaryngology, Head and Neck Surgery, School of Medicine, University of California, Davis, Sacramento, CA, United States.

ORCID profiles: Sandra L Taylor, Melissa Jost, Scott MacDonald, Yunyi Ren, Shelley Hilton, Sadie Davenport, Debbie Aizenberg, Bruce Hall, Courtney R Lyles, Jason Y Adams

Funding

  • NCATS NIH HHS

Cite

APA

Taylor, S. L., Jost, M., MacDonald, S., Ren, Y., Hilton, S., Davenport, S., Aizenberg, D., Hall, B., Lyles, C. R., & Adams, J. Y. (2026). Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study. JMIR Medical Informatics, 14, e86474-e86474. https://doi.org/10.2196/86474

Vancouver

Taylor SL, Jost M, MacDonald S, Ren Y, Hilton S, Davenport S, et al. Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study. JMIR Medical Informatics. 2026;14:e86474-e86474. doi:10.2196/86474

BibTeX
@article{Taylor2026,
  author = {Taylor, Sandra L and Jost, Melissa and MacDonald, Scott and Ren, Yunyi and Hilton, Shelley and Davenport, Sadie and Aizenberg, Debbie and Hall, Bruce and Lyles, Courtney R and Adams, Jason Y},
  title = {Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  volume = {14},
  pages = {e86474--e86474},
  doi = {10.2196/86474}
}
Start for free →