EUVIMEDOpdag EUVIMED
Crossref

Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth

Silvan Hornstein, Ulrike Lueken, Richard Wundrack, Hilbert, Kevin

edoc Publication server (Humboldt University of Berlin) · 2025-02-18

DOI: 10.18452/33052 ↗

At a glance

Journal article in edoc Publication server (Humboldt University of Berlin), published on 2025-02-18. Written by Silvan Hornstein and 4 co-authors.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Chat-based counseling services are popular for the low-threshold provision of mental health support to youth. In addition, they are particularly suitable for the utilization of natural language processing (NLP) for improved provision of care.

Objective

Consequently, this paper evaluates the feasibility of such a use case, namely, the NLP-based automated evaluation of satisfaction with the chat interaction. This preregistered approach could be used for evaluation and quality control procedures, as it is particularly relevant for those services.

Methods

The consultations of 2609 young chatters (around 140,000 messages) and corresponding feedback were used to train and evaluate classifiers to predict whether a chat was perceived as helpful or not. On the one hand, we trained a word vectorizer in combination with an extreme gradient boosting (XGBoost) classifier, applying cross-validation and extensive hyperparameter tuning. On the other hand, we trained several transformer-based models, comparing model types, preprocessing, and over- and undersampling techniques. For both model types, we selected the best-performing approach on the training set for a final performance evaluation on the 522 users in the final test set.

Results

The fine-tuned XGBoost classifier achieved an area under the receiver operating characteristic score of 0.69 (P

Subjects

chat counselingyouth mental healthuser satisfaction predictionperceived helpfulnessnatural language processingXGBoosttransformer modelsLongformerSHAPquality control

Cite

APA

Hornstein, S., Lueken, U., Wundrack, R., Hilbert, & Kevin (2025). Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth. edoc Publication server (Humboldt University of Berlin). https://doi.org/10.18452/33052

Vancouver

Hornstein S, Lueken U, Wundrack R, Hilbert, Kevin. Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth. edoc Publication server (Humboldt University of Berlin). 2025. doi:10.18452/33052

BibTeX
@article{Hornstein2025,
  author = {Hornstein, Silvan and Lueken, Ulrike and Wundrack, Richard and Hilbert and Kevin},
  title = {Predicting Satisfaction With Chat-Counseling at a 24/7 Chat Hotline for the Youth},
  journal = {edoc Publication server (Humboldt University of Berlin)},
  year = {2025},
  doi = {10.18452/33052}
}
Start for free →