Automated classification of exposure and encourage events in speech data from pediatric OCD treatment
JAMIA Open · 03.11.2025 · 8 (6)
DOI: 10.1093/jamiaopen/ooaf127 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 09.12.2025. Verfasst von Juan Antonio Lossio-Ventura und 10 weiteren Autor:innen. Beteiligte Einrichtung: Machine Learning Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIMH NIH HHS, National Institute of Mental Health, NIMH.
Abstract (Originalsprache)
Objective
To develop and evaluate an automated classification system for labeling Exposure Process Coding System (EPCS) quality codes—specifically exposure and encourage events—during in-person exposure therapy sessions using automatic speech recognition (ASR) and natural language processing techniques. Materials and Methods The system was trained and tested on 360 manually labeled pediatric Obsessive-Compulsive Disorder (OCD) therapy sessions from 3 clinical trials. Audio recordings were transcribed using ASR tools (OpenAI’s Whisper and Google Speech-to-Text). Transcription accuracy was evaluated via word error rate (WER) on manual transcriptions of 2-minute audio segments compared against ASR-generated transcripts. The resulting text was analyzed with transformer-based models, including Bidirectional Encoder Representations from Transformers (BERT), Sentence-BERT, and Meta Llama 3. Models were trained to predict EPCS codes in 2 classification settings: sequence-level classification, where events are labeled in delimited text chunks, and token-level classification, where event boundaries are unknown. Classification was performed either with fine-tuned transformer-based models, or with logistic regression on embeddings produced by each model.
Results
With respect to transcription accuracy, Whisper outperformed Google Speech-to-Text with a lower WER (0.31 vs 0.51). For sequence classification setting, Llama 3 models achieved high performance with area under the ROC curve (AUC) scores of 0.95 for exposures and 0.75 for encourage events, outperforming traditional methods and standard BERT models. In the token-level setting, fine-tuned BERT models performed best, achieving AUC scores of 0.85 for exposures and 0.75 for encourage events. Discussion and Conclusion Current ASR and transformer-based models enable automated quality coding of in-person exposure therapy sessions. These findings demonstrate potential for real-time assessment in clinical practice and scalable research on effective therapy methods. Future work should focus on optimization, including improvements in ASR accuracy, expanding training datasets, and multimodal data integration.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Machine Learning Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States.
- Emotion and Development Branch, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States.
- Department of Electrical and Biomedical Engineering, University of Vermont, Burlington, VT 05405, United States.
- Psychiatry and Human Behavior, Warren Alpert Medical School, Brown University, East Providence, RI 02915, United States.
Förderung
- NIMH NIH HHS
- National Institute of Mental Health
- NIMH
- Intramural Research Program of the National Institute of Mental Health
Zitieren
APA
Lossio-Ventura, J. A., Frank, S., Ringlein, G., Bonson, K., Olszko, A., Knobel, A., Pine, D. S., Freeman, J. B., Benito, K., Jangraw, D. C., & Pereira, F. (2025). Automated classification of exposure and encourage events in speech data from pediatric OCD treatment. JAMIA Open, 8(6). https://doi.org/10.1093/jamiaopen/ooaf127
Vancouver
Lossio-Ventura JA, Frank S, Ringlein G, Bonson K, Olszko A, Knobel A, et al. Automated classification of exposure and encourage events in speech data from pediatric OCD treatment. JAMIA Open. 2025;8(6). doi:10.1093/jamiaopen/ooaf127
BibTeX
@article{LossioVentura2025,
author = {Lossio-Ventura, Juan Antonio and Frank, Samuel and Ringlein, Grace and Bonson, Kirsten and Olszko, Ardyn and Knobel, Abbey and Pine, Daniel S and Freeman, Jennifer B and Benito, Kristen and Jangraw, David C and Pereira, Francisco},
title = {Automated classification of exposure and encourage events in speech data from pediatric OCD treatment},
journal = {JAMIA Open},
year = {2025},
volume = {8},
number = {6},
doi = {10.1093/jamiaopen/ooaf127}
}
EUVIMED