Benchmarking Transformer-Based Language Models for Mental Health Text Classification on Social Media
International Journal of Professional Practice · 17.11.2026 · 14 (9) · 26-35
DOI: 10.71274/ijpp.v14i9.780 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Professional Practice, online erschienen am 17.11.2026. Verfasst von David Muturi und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Mental disorders affect about 970 million people worldwide, yet few are ever screened in a clinic. Reddit posts contain people's own accounts of distress, making automated text classification an attractive, low-cost screening aid, but published results for competing transformer models use different datasets and protocols and cannot be compared. This study compared BERT, RoBERTa, MentalBERT, and DeBERTa-v3 under a single protocol to recommend a model for low-resource, multi-class screening. A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was combined with a benchmark on Reddit posts labelled by subreddit into four classes: anxiety, depression, bipolar disorder and schizophrenia. Each model was fine-tuned with identical hyperparameters and class-weighted cross-entropy loss on a stratified sample of 20,001 posts on one NVIDIA T4 graphics processing unit (GPU), then tested on 3,001 held-out posts using macro-F1, accuracy and per-class precision and recall. Stability came from 30 bootstrap resamples of the test predictions, pairwise differences were tested with McNemar's test, and a decision matrix weighted macro-F1 (30%), domain pretraining relevance (25%), stability (15%), literature evidence (15%), efficiency (10%) and pretraining data overlap (5%). RoBERTa-base had the highest bootstrap macro-F1 (0.806), with DeBERTa-v3-base close behind (0.805). MentalBERT did not differ significantly from RoBERTa-base or BERT-base (p > .05). The matrix ranked MentalBERT first (8.07/10), but a sensitivity analysis showed that this depends on the judged criteria. Domain-adaptive pretraining gave no measurable advantage at this scale. Model choice should weigh accuracy, cost and domain fit together, and outputs should serve only as a triage signal.
Zitieren
APA
Muturi, D., Nyale, D., & Shikali, C. (2026). Benchmarking Transformer-Based Language Models for Mental Health Text Classification on Social Media. International Journal of Professional Practice, 14(9), 26-35. https://doi.org/10.71274/ijpp.v14i9.780
Vancouver
Muturi D, Nyale D, Shikali C. Benchmarking Transformer-Based Language Models for Mental Health Text Classification on Social Media. International Journal of Professional Practice. 2026;14(9):26-35. doi:10.71274/ijpp.v14i9.780
BibTeX
@article{Muturi2026,
author = {Muturi, David and Nyale, Duncan and Shikali, Casper},
title = {Benchmarking Transformer-Based Language Models for Mental Health Text Classification on Social Media},
journal = {International Journal of Professional Practice},
year = {2026},
volume = {14},
number = {9},
pages = {26--35},
doi = {10.71274/ijpp.v14i9.780}
}
EUVIMED