EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Automating Systematic Reviews in Clinical Psychiatry: Comparing Domain Experts and NLP-Based Text Mining

Cyril S. Ku, Daniel Weiner, Meera Wells, Andrew Huang, Morgan R. Peltier

Information · 09.05.2026 · 17 (5) · 463

DOI: 10.3390/info17050463 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information, online erschienen am 09.05.2026. Verfasst von Cyril S. Ku und 4 weiteren Autor:innen. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

This study examines the potential of natural language processing and text mining to automate the systematic review process in clinical psychiatry, a field that traditionally relies on domain experts and can be time-consuming, prone to human bias and errors. The study compares the classification of review articles by domain experts with that facilitated by machine algorithms.

Methods

Using data from PubMed, 160 abstracts related to “transcranial magnetic stimulation” and “autism” were classified into “treatment” and “non-treatment” categories by both human reviewers and a computer algorithm. The computer algorithm, employing topic modeling in text mining, was compared to human reviewers, including two psychiatrists, a biostatistician, and a medical student.

Results

The accuracy of human classifications ranged from 68% to 85%, with inter-rater reliability (Kappa statistic) between 0.40 (fair to moderate) and 0.64 (substantial). Intra-rater reliability, tested by reclassification after three months, varied from 0.38 to 0.82.

Conclusions

The findings highlight the consistency and reproducibility of computational approaches compared to human classification, which exhibited both inter-rater and intra-rater variability. Differences in reviewer performance were observed; however, these patterns should be interpreted cautiously, as the study was not designed to directly assess cognitive or decision-making processes.

Förderung

  • National Science Foundation

Zitieren

APA

Ku, C. S., Weiner, D., Wells, M., Huang, A., & Peltier, M. R. (2026). Automating Systematic Reviews in Clinical Psychiatry: Comparing Domain Experts and NLP-Based Text Mining. Information, 17(5), 463. https://doi.org/10.3390/info17050463

Vancouver

Ku CS, Weiner D, Wells M, Huang A, Peltier MR. Automating Systematic Reviews in Clinical Psychiatry: Comparing Domain Experts and NLP-Based Text Mining. Information. 2026;17(5):463. doi:10.3390/info17050463

BibTeX
@article{Ku2026,
  author = {Ku, Cyril S. and Weiner, Daniel and Wells, Meera and Huang, Andrew and Peltier, Morgan R.},
  title = {Automating Systematic Reviews in Clinical Psychiatry: Comparing Domain Experts and NLP-Based Text Mining},
  journal = {Information},
  year = {2026},
  volume = {17},
  number = {5},
  pages = {463},
  doi = {10.3390/info17050463}
}
Kostenlos starten →