EUVIMEDEUVIMED entdecken
Open Access

Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study

Sandra L Taylor iD, Melissa Jost iD, Scott MacDonald iD, Yunyi Ren iD, Shelley Hilton iD, Sadie Davenport iD, Debbie Aizenberg iD, Bruce Hall iD, Courtney R Lyles iD, Jason Y Adams iD

JMIR Medical Informatics · 17.04.2026 · 14 · e86474-e86474

DOI: 10.2196/86474 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, online erschienen am 17.04.2026. Verfasst von Sandra L Taylor und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Public Health Sciences, School of Medicine, University of California, Davis, 4480 2nd Avenue, Suite 4152, Sacramento, CA, 95817, United…. Hauptthemen laut MEDLINE: Documentation, Electronic Health Records, Artificial Intelligence. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Physicians routinely document specifics of patient encounters in clinic visit notes, a critical but potentially time-consuming task. Ambient listening artificial intelligence (AI) technology is being integrated into clinical workflows to reduce documentation burden by creating draft visit notes. While this technology is promising, it is not perfect, and the potential for patient harm needs to be understood and mitigated. We developed and piloted an efficient, standardized approach to evaluating AI-generated notes for safety concerns in ambulatory care visits.

Objective

The objective of this quality improvement project was to develop and pilot an efficient, standardized, and scalable approach to evaluating AI-generated notes for safety concerns in ambulatory care visits.

Methods

During a 2-month pilot (July to August 2024), 31 physicians across multiple specialties used an ambient listening AI scribe to assist with the creation of 7545 clinic notes. A novel survey instrument was developed to assess note quality, focusing on 4 error types: accidental inclusions, accidental omissions, hallucinations, and bias. Physicians evaluated 356 (4.7%) AI-generated notes. Where an error was present, physicians rated its severity based on its potential to cause patient harm if it was not corrected, on a 0 to 5 scale. Additionally, a vendor-reported metric on the percentage of note content edited by physicians was analyzed.

Results

Of the 356 evaluated notes, accidental omissions were the most frequent error (n=64, 18%), followed by hallucinations (n=41, 11.5%), and accidental inclusions (n=33, 9.3%). Bias was rare (n=4, 1.1%). Most (119/142, 83.8%) errors were rated as mild to moderate (severity 1‐3), with only 19 (5.3%) notes containing errors rated as posing serious or imminent risk (severity 4‐5). Editing metrics across all AI-created notes showed a median of 9.0% (IQR 2.5%-21.9%) of AI-generated words were changed, with 14.9% (143/960) of notes left entirely unedited. Physician editing practices varied widely, with average percentages of AI-generated words changed ranging from 1.9% to 69.3% (median 9.0%, IQR 2.5%-21.9%).

Conclusions

AI-generated clinical notes were generally of high quality, with 94.7% (337/356) free from significant errors. However, because a small number contained errors that carried the risk of serious harm if not corrected, careful clinician review of notes remains imperative. Prior to deploying an AI scribe, organizations should pilot the technology and include an efficient review process to understand the nature and type of errors common at their organization. This pilot provides a scalable model for other health systems seeking to implement AI scribe technology responsibly.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DocumentationElectronic Health RecordsArtificial IntelligencePilot ProjectsHumansProspective StudiesGenerative Artificial IntelligenceQuality ImprovementSurveys and Questionnaires

Stichwörter der Autor:innen

AI scribeambient listeningartificial intelligence scribeclinical documentationgenerative AIgenerative artificial intelligencequality of care

Autor:innen und Institutionen

  • Department of Public Health Sciences, School of Medicine, University of California, Davis, 4480 2nd Avenue, Suite 4152, Sacramento, CA, 95817, United States, 1 916-734-4800.
  • Clinical and Translational Science Center, University of California, Davis, Sacramento, CA, United States.
  • Department of Clinical Informatics, University of California, Davis, Sacramento, CA, United States.
  • IT Clinical Applications, University of California, Davis, Sacramento, CA, United States.
  • Blaisdell Medical Library, University of California, Davis, Sacramento, CA, United States.
  • Department of Otolaryngology, Head and Neck Surgery, School of Medicine, University of California, Davis, Sacramento, CA, United States.

ORCID-Profile: Sandra L Taylor, Melissa Jost, Scott MacDonald, Yunyi Ren, Shelley Hilton, Sadie Davenport, Debbie Aizenberg, Bruce Hall, Courtney R Lyles, Jason Y Adams

Förderung

  • NCATS NIH HHS

Zitieren

APA

Taylor, S. L., Jost, M., MacDonald, S., Ren, Y., Hilton, S., Davenport, S., Aizenberg, D., Hall, B., Lyles, C. R., & Adams, J. Y. (2026). Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study. JMIR Medical Informatics, 14, e86474-e86474. https://doi.org/10.2196/86474

Vancouver

Taylor SL, Jost M, MacDonald S, Ren Y, Hilton S, Davenport S, et al. Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study. JMIR Medical Informatics. 2026;14:e86474-e86474. doi:10.2196/86474

BibTeX
@article{Taylor2026,
  author = {Taylor, Sandra L and Jost, Melissa and MacDonald, Scott and Ren, Yunyi and Hilton, Shelley and Davenport, Sadie and Aizenberg, Debbie and Hall, Bruce and Lyles, Courtney R and Adams, Jason Y},
  title = {Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  volume = {14},
  pages = {e86474--e86474},
  doi = {10.2196/86474}
}
Kostenlos starten →