Temporal and cross-site validation of an AI system for self-harm detection
PLOS Digital Health · 11.09.2026 · 5 (9) · e0001667
DOI: 10.1371/journal.pdig.0001667 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 11.09.2026. Verfasst von Vlada Rozova und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Data Science and AI, Monash University, Melbourne, Australia. Die Arbeit nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Health and Medical Research Council, Department of Health, State Government of Victoria, William Buckland Foundation.
Abstract (Originalsprache)
Adequate self-harm surveillance is a key part of suicide prevention. Our previous research demonstrated that an artificial intelligence (AI)-based system could effectively detect self-harm in emergency department triage notes. However, the system was developed using data from a single hospital, raising concerns about its generalisability. Here, we aim to validate the system prospectively and externally to better understand its portability across hospitals. We leveraged emergency department data from two Australian hospitals, with free-text triage notes manually annotated for self-harm. The AI system was developed using data from a major metropolitan hospital in Melbourne, spanning 2012–2017. The system combined extensive text normalisation with a self-harm classification model utilising 1931 selected features. We prospectively validated the model on 329,655 triage notes from the same hospital over the following four years. For external validation, we used 316,877 triage notes from 2012 to 2021 from a regional hospital 150 km outside Melbourne. On the test set, the model achieved an area under the precision-recall curve (PR AUC) of 0.84 with a 95% confidence interval of [0.82, 0.86]. This performance remained stable at the development site with a PR AUC of 0.84, 95% CI [0.83, 0.85]. When applied in the regional context, the model’s ability to distinguish self-harm cases declined, resulting in an overall PR AUC of 0.78, 95% CI [0.77, 0.79]. While the text normalisation component was equally effective across the datasets, our analysis revealed that in regional settings, self-harm presentations are more likely to involve medication ingestion. At the metropolitan hospital, the AI system for self-harm detection maintained its epidemiological utility. At the regional hospital, the text normalisation process was effective, but performance was unstable primarily due to linguistic domain shift and differences in self-harm presentations.
Autor:innen und Institutionen
- Department of Data Science and AI, Monash University, Melbourne, Australia.
- School of Computing and Information Systems, University of Melbourne, Melbourne, Australia.
- Centre for Digital Transformation of Health, University of Melbourne, Melbourne, Australia.
- Centre for Youth Mental Health, The University of Melbourne, Melbourne, Australia.
- Orygen, Melbourne, Australia.
- School of Computing Technologies, RMIT University, Melbourne, Australia.
ORCID-Profile: Vlada Rozova, Karin Verspoor
Förderung
- National Health and Medical Research Council
- Department of Health, State Government of Victoria
- William Buckland Foundation
- Future Generation Global
- Medical Research Future Fund
Zitieren
APA
Rozova, V., Chen, L., Witt, K., Conway, M., Robinson, J., & Verspoor, K. (2026). Temporal and cross-site validation of an AI system for self-harm detection. PLOS Digital Health, 5(9), e0001667. https://doi.org/10.1371/journal.pdig.0001667
Vancouver
Rozova V, Chen L, Witt K, Conway M, Robinson J, Verspoor K. Temporal and cross-site validation of an AI system for self-harm detection. PLOS Digital Health. 2026;5(9):e0001667. doi:10.1371/journal.pdig.0001667
BibTeX
@article{Rozova2026,
author = {Rozova, Vlada and Chen, Liuliu and Witt, Katrina and Conway, Mike and Robinson, Jo and Verspoor, Karin},
title = {Temporal and cross-site validation of an AI system for self-harm detection},
journal = {PLOS Digital Health},
year = {2026},
volume = {5},
number = {9},
pages = {e0001667},
doi = {10.1371/journal.pdig.0001667}
}
EUVIMED