Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth
edoc Publication server (Humboldt University of Berlin) · 2025-02-18
DOI: 10.18452/33052 ↗At a glance
Journal article in edoc Publication server (Humboldt University of Berlin), published on 2025-02-18. Written by Silvan Hornstein and 4 co-authors.
Abstract (original language)
Background
Chat-based counseling services are popular for the low-threshold provision of mental health support to youth. In addition, they are particularly suitable for the utilization of natural language processing (NLP) for improved provision of care.
Objective
Consequently, this paper evaluates the feasibility of such a use case, namely, the NLP-based automated evaluation of satisfaction with the chat interaction. This preregistered approach could be used for evaluation and quality control procedures, as it is particularly relevant for those services.
Methods
The consultations of 2609 young chatters (around 140,000 messages) and corresponding feedback were used to train and evaluate classifiers to predict whether a chat was perceived as helpful or not. On the one hand, we trained a word vectorizer in combination with an extreme gradient boosting (XGBoost) classifier, applying cross-validation and extensive hyperparameter tuning. On the other hand, we trained several transformer-based models, comparing model types, preprocessing, and over- and undersampling techniques. For both model types, we selected the best-performing approach on the training set for a final performance evaluation on the 522 users in the final test set.
Results
The fine-tuned XGBoost classifier achieved an area under the receiver operating characteristic score of 0.69 (P
Subjects
Cite
APA
Hornstein, S., Lueken, U., Wundrack, R., Hilbert, & Kevin (2025). Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth. edoc Publication server (Humboldt University of Berlin). https://doi.org/10.18452/33052
Vancouver
Hornstein S, Lueken U, Wundrack R, Hilbert, Kevin. Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth. edoc Publication server (Humboldt University of Berlin). 2025. doi:10.18452/33052
BibTeX
@article{Hornstein2025,
author = {Hornstein, Silvan and Lueken, Ulrike and Wundrack, Richard and Hilbert and Kevin},
title = {Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth},
journal = {edoc Publication server (Humboldt University of Berlin)},
year = {2025},
doi = {10.18452/33052}
}
EUVIMED