Machine Learning–Based Evaluation of Suicide Risk Assessment in Crisis Counseling Calls
Psychiatric Services · 01.11.2024 · 75 (11) · 1068-1074
DOI: 10.1176/appi.ps.20230648 ↗Kurz gefasst
Fachartikel in der Zeitschrift Psychiatric Services, erschienen am 01.11.2024. Verfasst von Zac E Imel und 7 weiteren Autor:innen. Beteiligte Einrichtung: Lyssn.io, Seattle (Imel, Pace, Pruett, Tanana, Soma, Atkins); Protocall Services, Portland, Oregon (Pendergraft); Harborview Medical Center, Universi…. Hauptthemen laut MEDLINE: Machine Learning, Counseling. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIMH NIH HHS.
Abstract (Originalsprache)
OBJECTIVE: Counselor assessment of suicide risk is one key component of crisis counseling, and standards require risk assessment in every crisis counseling conversation. Efforts to increase risk assessment frequency are limited by quality improvement tools that rely on human evaluation of conversations, which is labor intensive, slow, and impossible to scale. Advances in machine learning (ML) have made possible the development of tools that can automatically and immediately detect the presence of risk assessment in crisis counseling conversations. METHODS: To train models, a coding team labeled every statement in 476 crisis counseling calls (193,257 statements) for a core element of risk assessment. The authors then fine-tuned a transformer-based ML model with the labeled data, utilizing separate training, validation, and test data sets. RESULTS: Generally, the evaluated ML model was highly consistent with human raters. For detecting any risk assessment, ML model agreement with human ratings was 98% of human interrater agreement. Across specific labels, average F1 (the harmonic mean of precision and recall) was 0.86 at the call level and 0.66 at the statement level and often varied as a result of a low base rate for some risk labels. CONCLUSIONS: ML models can reliably detect the presence of suicide risk assessment in crisis counseling conversations, presenting an opportunity to scale quality improvement efforts.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Lyssn.io, Seattle (Imel, Pace, Pruett, Tanana, Soma, Atkins); Protocall Services, Portland, Oregon (Pendergraft); Harborview Medical Center, University of Washington, Seattle (Comtois).
Förderung
- NIMH NIH HHS
Zitieren
APA
Imel, Z. E., Pace, B., Pendergraft, B., Pruett, J., Tanana, M., Soma, C. S., Comtois, K. A., & Atkins, D. C. (2024). Machine Learning–Based Evaluation of Suicide Risk Assessment in Crisis Counseling Calls. Psychiatric Services, 75(11), 1068-1074. https://doi.org/10.1176/appi.ps.20230648
Vancouver
Imel ZE, Pace B, Pendergraft B, Pruett J, Tanana M, Soma CS, et al. Machine Learning–Based Evaluation of Suicide Risk Assessment in Crisis Counseling Calls. Psychiatric Services. 2024;75(11):1068-1074. doi:10.1176/appi.ps.20230648
BibTeX
@article{Imel2024,
author = {Imel, Zac E and Pace, Brian and Pendergraft, Brad and Pruett, Jordan and Tanana, Michael and Soma, Christina S and Comtois, Kate A and Atkins, David C},
title = {Machine Learning–Based Evaluation of Suicide Risk Assessment in Crisis Counseling Calls},
journal = {Psychiatric Services},
year = {2024},
volume = {75},
number = {11},
pages = {1068--1074},
doi = {10.1176/appi.ps.20230648}
}
EUVIMED