Automating Systematic Reviews in Clinical Psychiatry: Comparing Domain Experts and NLP-Based Text Mining
Information · 09.05.2026 · 17 (5) · 463
DOI: 10.3390/info17050463 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information, online erschienen am 09.05.2026. Verfasst von Cyril S. Ku und 4 weiteren Autor:innen. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.
Abstract (Originalsprache)
Objective
This study examines the potential of natural language processing and text mining to automate the systematic review process in clinical psychiatry, a field that traditionally relies on domain experts and can be time-consuming, prone to human bias and errors. The study compares the classification of review articles by domain experts with that facilitated by machine algorithms.
Methods
Using data from PubMed, 160 abstracts related to “transcranial magnetic stimulation” and “autism” were classified into “treatment” and “non-treatment” categories by both human reviewers and a computer algorithm. The computer algorithm, employing topic modeling in text mining, was compared to human reviewers, including two psychiatrists, a biostatistician, and a medical student.
Results
The accuracy of human classifications ranged from 68% to 85%, with inter-rater reliability (Kappa statistic) between 0.40 (fair to moderate) and 0.64 (substantial). Intra-rater reliability, tested by reclassification after three months, varied from 0.38 to 0.82.
Conclusions
The findings highlight the consistency and reproducibility of computational approaches compared to human classification, which exhibited both inter-rater and intra-rater variability. Differences in reviewer performance were observed; however, these patterns should be interpreted cautiously, as the study was not designed to directly assess cognitive or decision-making processes.
Förderung
- National Science Foundation
Zitieren
APA
Ku, C. S., Weiner, D., Wells, M., Huang, A., & Peltier, M. R. (2026). Automating Systematic Reviews in Clinical Psychiatry: Comparing Domain Experts and NLP-Based Text Mining. Information, 17(5), 463. https://doi.org/10.3390/info17050463
Vancouver
Ku CS, Weiner D, Wells M, Huang A, Peltier MR. Automating Systematic Reviews in Clinical Psychiatry: Comparing Domain Experts and NLP-Based Text Mining. Information. 2026;17(5):463. doi:10.3390/info17050463
BibTeX
@article{Ku2026,
author = {Ku, Cyril S. and Weiner, Daniel and Wells, Meera and Huang, Andrew and Peltier, Morgan R.},
title = {Automating Systematic Reviews in Clinical Psychiatry: Comparing Domain Experts and NLP-Based Text Mining},
journal = {Information},
year = {2026},
volume = {17},
number = {5},
pages = {463},
doi = {10.3390/info17050463}
}
EUVIMED