AI-first, expert-verified: validating generative AI for HFACS-based coding of healthcare RCA transcripts with governance considerations
International Journal for Quality in Health Care · 13.04.2026 · 38 (2)
DOI: 10.1093/intqhc/mzag045 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift International Journal for Quality in Health Care, online erschienen am 09.04.2026. Verfasst von Jiun-Yih Lee und 8 weiteren Autor:innen. Beteiligte Einrichtung: Center for Quality Management, Shin Kong Wu Ho-Su Memorial Hospital, Taipei, Taiwan. Hauptthemen laut MEDLINE: Generative Artificial Intelligence, Root Cause Analysis, Artificial Intelligence, Clinical Coding. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 13 Literaturangaben. Gefördert von: Shin Kong Wu Ho-Su Memorial Hospital.
Abstract (Originalsprache)
Background
Root cause analysis (RCA) is widely used in healthcare incident investigation, but its outputs can be limited by inconsistent causal framing and variable integration of human factors. The Human Factors Analysis and Classification System (HFACS) offers a structured taxonomy for causal attribution, yet manual coding is resource-intensive. Empirical validation of generative artificial intelligence (AI) for document-level HFACS coding from complete healthcare RCA transcripts remains limited.
Methods
We conducted a cross-sectional validation study at an 829-bed medical center in Taiwan. Thirty-five de-identified RCA interview transcripts (2024–5) with verbatim transcription were analyzed using SKH-AI, an in-house platform integrating an Azure OpenAI–hosted GPT-4o model with deterministic decoding (temperature = 0; top_p = 1.0). The model processed each transcript holistically to identify salient narrative segments and assign HFACS codes with evidence-linked rationales without rule-based post-processing. Outputs were compared with dual-expert HFACS coding with adjudicated consensus. Performance was assessed using precision, recall, micro-/macro-F1, and Cohen’s κ with bootstrapped 95% confidence intervals.
Results
Across 562 AI-derived segments, micro-F1 was 0.66 [95% confidence interval (CI): 0.63–0.69] and macro-F1 was 0.68 (95% CI: 0.64–0.72), with moderate agreement versus expert coding (κ = 0.56, 95% CI: 0.52–0.60). Performance was higher for text-anchored categories (Level 2 Preconditions F1 = 0.70; Level 1 Unsafe Acts F1 = 0.69) than for more abstract domains (Level 3 F1 = 0.66; Level 4 F1 = 0.65). Subcategory analyses showed stronger detection of concrete cues (e.g. decision errors, physical environment, communication) and weaker performance for latent constructs (e.g. process management). Bias analyses indicated a recall-leaning tendency at Levels 3–4, consistent with increased over-attribution risk.
Conclusions
Generative AI can produce auditable, evidence-linked candidate HFACS attributions from document-level RCA transcripts with moderate concordance to expert coding. Higher-level supervisory and organizational attributions remain vulnerable to overgeneralization and should be governed as decision support, with evidence-anchoring and mandatory expert sign-off for Level 3–4 codes.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Quality Management, Shin Kong Wu Ho-Su Memorial Hospital, Taipei, Taiwan.
- Division of Infectious Disease, Department of Internal Medicine, Shin Kong Wu Ho-Su Memorial Hospital, Taipei, Taiwan.
- College of Medicine, Fu Jen Catholic University, New Taipei, Taiwan.
- Department of General Surgery, Shin Kong Wu Ho-Su Memorial Hospital, Taipei, Taiwan.
- College of Science and Engineering, Fu Jen Catholic University, New Taipei City, Taiwan.
- Department of Information Technology, Shin Kong Wu Ho-Su Memorial Hospital, Taipei, Taiwan.
ORCID-Profile: Jiun-Yih Lee
Förderung
- Shin Kong Wu Ho-Su Memorial Hospital
Zitieren
APA
Lee, J. Y., Huang, C. H., Hsieh, C. H., Lin, C. S., Yang, P. C., Lai, Y. L., Yeh, C. A., Tien, C. J., & Chang, J. T. (2026). AI-first, expert-verified: validating generative AI for HFACS-based coding of healthcare RCA transcripts with governance considerations. International Journal for Quality in Health Care, 38(2). https://doi.org/10.1093/intqhc/mzag045
Vancouver
Lee JY, Huang CH, Hsieh CH, Lin CS, Yang PC, Lai YL, et al. AI-first, expert-verified: validating generative AI for HFACS-based coding of healthcare RCA transcripts with governance considerations. International Journal for Quality in Health Care. 2026;38(2). doi:10.1093/intqhc/mzag045
BibTeX
@article{Lee2026,
author = {Lee, Jiun-Yih and Huang, Chien-Hsien and Hsieh, Chung-Ho and Lin, Chin-Shu and Yang, Pei-Ching and Lai, Yan-Liang and Yeh, Chen-An and Tien, Chin-Jui and Chang, Jui-Ting},
title = {AI-first, expert-verified: validating generative AI for HFACS-based coding of healthcare RCA transcripts with governance considerations},
journal = {International Journal for Quality in Health Care},
year = {2026},
volume = {38},
number = {2},
doi = {10.1093/intqhc/mzag045}
}
EUVIMED