EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors

Die Effekte künstlicher Intelligenz auf die Qualitätsbewertung ambulanter psychiatrischer Leistungen durch menschliche Auditor:innen

Hayden T McLeod, Joshua L Cohen, Dena M Bravata iD, Roma M Bajaria, Kelvin Tran, Neha P Chaudhary, Nicole M Benson iD, Jeff Gould

Applied Clinical Informatics · 01.08.2026 · 17 (04) · 834-843

DOI: 10.1055/a-2961-0302 ↗

Auf einen Blick

Eine Vorher-nachher-Beobachtungsstudie untersuchte LLM-gestützte Qualitätsbewertungen von Dokumentationen ambulanter psychiatrischer Intake-Sitzungen. Nach Einführung der KI stieg die mittlere Passrate leicht, und die Bearbeitungszeit sank.

Fragestellung

Wie beeinflusst die Exposition gegenüber LLM-generierten Qualitätsbewertungen die Leistung von Auditor:innen bei der Prüfung ambulanter psychiatrischer Intake-Dokumente?

Methode

Vorher-nachher-Beobachtung mit 10 geschulten menschlichen Auditor:innen; Bewertung von Intake-Notizen mit einem standardisierten 13-Item-Chart-Audit-Raster vor und nach LLM-Unterstützung.

Ergebnisse

Vor KI: 9.711 Notizen (126.243 Item-Observations); nach KI: 2.677 Notizen (34.801 Item-Observations). Mittlere Item-Passrate um etwa 2 % höher; Reviewzeit 8,77 min vs. 8,06 min; p = 0,002.

Schlussfolgerung der Autor:innen

LLM-basierte KI-Tools können die Effizienz klinischer Qualitätsprüfungen verbessern, die KI-Empfehlungen können das Verhalten menschlicher Auditor:innen jedoch relevant verändern.

Kernaussagen

  • Mittlere Item-Passrate nach KI-Einsatz um etwa 2 % höher
  • Reviewzeit sank von 8,77 auf 8,06 min pro Prüfung
  • Deutliche Heterogenität der individuellen Reaktionen auf KI-Unterstützung

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Fachartikel in der Zeitschrift Applied Clinical Informatics, erschienen am 01.08.2026. Verfasst von Hayden T McLeod und 7 weiteren Autor:innen. Beteiligte Einrichtung: TherapyMatch, Inc (dba Headway), New York, United States, New York. Hauptthemen laut MEDLINE: Artificial Intelligence, Mental Health Services, Quality Assurance, Health Care, Outpatients. Die Arbeit nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: TherapyMatch.

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The use of large language model (LLM)-based artificial intelligence (AI) to assess the quality of outpatient mental health documentation represents an understudied application. Critical questions remain about the effects of exposing human auditors to AI-generated recommendations on auditor efficiency, agreement, and patterns of decision-making.

Objectives

The objective of this study is to examine how exposure to LLM-generated quality assessments influences auditor performance during reviews of documents of outpatient mental health intake sessions.

Methods

We conducted a pre–post observational evaluation of 10 trained human auditors who reviewed outpatient mental health intake notes and assigned quality ratings using a standardized 13-item chart audit rubric before and after implementation of LLM support. The AI system used Anthropic Claude Sonnet 4 with a fixed chain-prompting strategy. We assessed item-level pass frequency for each auditor across the pre-AI and post-AI periods, AI–human agreement (the alignment between auditor pass/fail determinations and AI-generated recommendations), and auditing time. The pre-AI period included 9,711 notes (126,243 item-level observations) and the post-AI period included 2,677 notes (34,801 item-level observations).

Results

Following implementation of AI-supported auditing, the mean item-level pass rate increased by approximately 2%. The distribution of auditor-level changes was broad (range: −35 to +35%, standard deviation [SD]: 10%) indicating substantial heterogeneity in individual auditor responses to AI support. High auditor agreement with AI recommendations was associated with larger pass-rate changes for selected rubric items, with some auditors increasing and others decreasing pass rates relative to the pre-AI period. The average time for an auditor to review notes decreased from 8.77 min (SD: 9.64) per review during the pre-AI period to 8.06 min (SD: 10.28) during the post-AI period (p = 0.002), an 8% improvement equivalent to 71 min saved per 100 notes reviewed.

Conclusion

LLM-based AI tools may improve the efficiency of clinical quality assurance workflows, but exposure to AI recommendations may meaningfully change human auditor behavior.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceMental Health ServicesQuality Assurance, Health CareOutpatientsHumansLarge Language Models

Autor:innen und Institutionen

  • TherapyMatch, Inc (dba Headway), New York, United States, New York.
  • Stanford University, Center for Primary Care and Outcomes Research, California, United States, San Mateo.
  • Massachusetts General Hospital, Harvard Medical School, Massachusetts, United States, Boston.
  • McLean Hospital, Harvard Medical School, Massachusetts, United States, Belmont.
  • University of California, California, United States, San Francisco.

ORCID-Profile: Dena M Bravata, Nicole M Benson

Förderung

  • TherapyMatch

Zitieren

APA

McLeod, H. T., Cohen, J. L., Bravata, D. M., Bajaria, R. M., Tran, K., Chaudhary, N. P., Benson, N. M., & Gould, J. (2026). The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors. Applied Clinical Informatics, 17(04), 834-843. https://doi.org/10.1055/a-2961-0302

Vancouver

McLeod HT, Cohen JL, Bravata DM, Bajaria RM, Tran K, Chaudhary NP, et al. The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors. Applied Clinical Informatics. 2026;17(04):834-843. doi:10.1055/a-2961-0302

BibTeX
@article{McLeod2026,
  author = {McLeod, Hayden T and Cohen, Joshua L and Bravata, Dena M and Bajaria, Roma M and Tran, Kelvin and Chaudhary, Neha P and Benson, Nicole M and Gould, Jeff},
  title = {The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors},
  journal = {Applied Clinical Informatics},
  year = {2026},
  volume = {17},
  number = {04},
  pages = {834--843},
  doi = {10.1055/a-2961-0302}
}
Kostenlos starten →