The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors
Die Effekte künstlicher Intelligenz auf die Qualitätsbewertung ambulanter psychiatrischer Leistungen durch menschliche Auditor:innen
Applied Clinical Informatics · 01.08.2026 · 17 (04) · 834-843
DOI: 10.1055/a-2961-0302 ↗Auf einen Blick
Eine Vorher-nachher-Beobachtungsstudie untersuchte LLM-gestützte Qualitätsbewertungen von Dokumentationen ambulanter psychiatrischer Intake-Sitzungen. Nach Einführung der KI stieg die mittlere Passrate leicht, und die Bearbeitungszeit sank.
Fragestellung
Wie beeinflusst die Exposition gegenüber LLM-generierten Qualitätsbewertungen die Leistung von Auditor:innen bei der Prüfung ambulanter psychiatrischer Intake-Dokumente?
Methode
Vorher-nachher-Beobachtung mit 10 geschulten menschlichen Auditor:innen; Bewertung von Intake-Notizen mit einem standardisierten 13-Item-Chart-Audit-Raster vor und nach LLM-Unterstützung.
Ergebnisse
Vor KI: 9.711 Notizen (126.243 Item-Observations); nach KI: 2.677 Notizen (34.801 Item-Observations). Mittlere Item-Passrate um etwa 2 % höher; Reviewzeit 8,77 min vs. 8,06 min; p = 0,002.
Schlussfolgerung der Autor:innen
LLM-basierte KI-Tools können die Effizienz klinischer Qualitätsprüfungen verbessern, die KI-Empfehlungen können das Verhalten menschlicher Auditor:innen jedoch relevant verändern.
Kernaussagen
- Mittlere Item-Passrate nach KI-Einsatz um etwa 2 % höher
- Reviewzeit sank von 8,77 auf 8,06 min pro Prüfung
- Deutliche Heterogenität der individuellen Reaktionen auf KI-Unterstützung
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Fachartikel in der Zeitschrift Applied Clinical Informatics, erschienen am 01.08.2026. Verfasst von Hayden T McLeod und 7 weiteren Autor:innen. Beteiligte Einrichtung: TherapyMatch, Inc (dba Headway), New York, United States, New York. Hauptthemen laut MEDLINE: Artificial Intelligence, Mental Health Services, Quality Assurance, Health Care, Outpatients. Die Arbeit nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: TherapyMatch.
Themen: Künstliche Intelligenz in der Pflege, Psychiatrische Pflege
Abstract (Originalsprache)
Background
The use of large language model (LLM)-based artificial intelligence (AI) to assess the quality of outpatient mental health documentation represents an understudied application. Critical questions remain about the effects of exposing human auditors to AI-generated recommendations on auditor efficiency, agreement, and patterns of decision-making.
Objectives
The objective of this study is to examine how exposure to LLM-generated quality assessments influences auditor performance during reviews of documents of outpatient mental health intake sessions.
Methods
We conducted a pre–post observational evaluation of 10 trained human auditors who reviewed outpatient mental health intake notes and assigned quality ratings using a standardized 13-item chart audit rubric before and after implementation of LLM support. The AI system used Anthropic Claude Sonnet 4 with a fixed chain-prompting strategy. We assessed item-level pass frequency for each auditor across the pre-AI and post-AI periods, AI–human agreement (the alignment between auditor pass/fail determinations and AI-generated recommendations), and auditing time. The pre-AI period included 9,711 notes (126,243 item-level observations) and the post-AI period included 2,677 notes (34,801 item-level observations).
Results
Following implementation of AI-supported auditing, the mean item-level pass rate increased by approximately 2%. The distribution of auditor-level changes was broad (range: −35 to +35%, standard deviation [SD]: 10%) indicating substantial heterogeneity in individual auditor responses to AI support. High auditor agreement with AI recommendations was associated with larger pass-rate changes for selected rubric items, with some auditors increasing and others decreasing pass rates relative to the pre-AI period. The average time for an auditor to review notes decreased from 8.77 min (SD: 9.64) per review during the pre-AI period to 8.06 min (SD: 10.28) during the post-AI period (p = 0.002), an 8% improvement equivalent to 71 min saved per 100 notes reviewed.
Conclusion
LLM-based AI tools may improve the efficiency of clinical quality assurance workflows, but exposure to AI recommendations may meaningfully change human auditor behavior.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- TherapyMatch, Inc (dba Headway), New York, United States, New York.
- Stanford University, Center for Primary Care and Outcomes Research, California, United States, San Mateo.
- Massachusetts General Hospital, Harvard Medical School, Massachusetts, United States, Boston.
- McLean Hospital, Harvard Medical School, Massachusetts, United States, Belmont.
- University of California, California, United States, San Francisco.
ORCID-Profile: Dena M Bravata, Nicole M Benson
Förderung
- TherapyMatch
Zitieren
APA
McLeod, H. T., Cohen, J. L., Bravata, D. M., Bajaria, R. M., Tran, K., Chaudhary, N. P., Benson, N. M., & Gould, J. (2026). The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors. Applied Clinical Informatics, 17(04), 834-843. https://doi.org/10.1055/a-2961-0302
Vancouver
McLeod HT, Cohen JL, Bravata DM, Bajaria RM, Tran K, Chaudhary NP, et al. The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors. Applied Clinical Informatics. 2026;17(04):834-843. doi:10.1055/a-2961-0302
BibTeX
@article{McLeod2026,
author = {McLeod, Hayden T and Cohen, Joshua L and Bravata, Dena M and Bajaria, Roma M and Tran, Kelvin and Chaudhary, Neha P and Benson, Nicole M and Gould, Jeff},
title = {The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors},
journal = {Applied Clinical Informatics},
year = {2026},
volume = {17},
number = {04},
pages = {834--843},
doi = {10.1055/a-2961-0302}
}
EUVIMED