StudyTypeTeller—Large language models to automatically classify research study types for systematic reviews
Research Synthesis Methods · 2025
DOI: 10.1017/rsm.2025.10031 ↗Kurz gefasst
Fachartikel in der Zeitschrift Research Synthesis Methods, 2025. Verfasst von Simona Emilova Doneva und 5 weiteren Autor:innen. Beteiligte Einrichtung: Center for Reproducible Science, https://ror.org/02crff812University of Zurich, Zurich, Switzerland. Hauptthemen laut MEDLINE: Large Language Models, Systematic Reviews as Topic, Natural Language Processing. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universität Zürich, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung.
Abstract (Originalsprache)
Abstract screening, a labor-intensive aspect of systematic review, is increasingly challenging due to the rising volume of scientific publications. Recent advances suggest that generative large language models like generative pre-trained transformer (GPT) could aid this process by classifying references into study types such as randomized-controlled trials (RCTs) or animal studies prior to abstract screening. However, it is unknown how these GPT models perform in classifying such scientific study types in the biomedical field. Additionally, their performance has not been directly compared with earlier transformer-based models like bidirectional encoder representations from transformers (BERT). To address this, we developed a human-annotated corpus of 2,645 PubMed titles and abstracts, annotated for 14 study types, including different types of RCTs and animal studies, systematic reviews, study protocols, case reports, as well as in vitro studies. Using this corpus, we compared the performance of GPT-3.5 and GPT-4 in automatically classifying these study types against established BERT models. Our results show that fine-tuned pretrained BERT models consistently outperformed GPT models, achieving F1-scores above 0.8, compared to approximately 0.6 for GPT models. Advanced prompting strategies did not substantially boost GPT performance. In conclusion, these findings highlight that, even though GPT models benefit from advanced capabilities and extensive training data, their performance in niche tasks like scientific multi-class study classification is inferior to smaller fine-tuned models. Nevertheless, the use of automated methods remains promising for reducing the volume of records, making the screening of large reference libraries more feasible. Our corpus is openly available and can be used to harness other natural language processing (NLP) approaches.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Reproducible Science, https://ror.org/02crff812University of Zurich, Zurich, Switzerland.
- CLEAR Methods Center, Division of Clinical Epidemiology, Department of Clinical Research, University Hospital Basel, https://ror.org/02s6k3f65University of Basel, Basel, Switzerland.
- MTA-PTE Lendület "Momentum" Evidence in Medicine Research Group, Medical School, https://ror.org/037b5pv06University of Pécs, Pécs, Hungary.
- Department of Health Research Methods, Evidence, and Impact, https://ror.org/02fa3aq29McMaster University, Hamilton, ON, Canada.
- Department of Clinical Research, https://ror.org/02k7v4d05University of Bern, Bern, Switzerland.
Förderung
- Universität Zürich
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
Zitieren
APA
Emilova Doneva, S., de Viragh, S., Hubarava, H., Schandelmaier, S., Briel, M., & Ineichen, B. V. (2025). StudyTypeTeller—Large language models to automatically classify research study types for systematic reviews. Research Synthesis Methods. https://doi.org/10.1017/rsm.2025.10031
Vancouver
Emilova Doneva S, de Viragh S, Hubarava H, Schandelmaier S, Briel M, Ineichen BV. StudyTypeTeller—Large language models to automatically classify research study types for systematic reviews. Research Synthesis Methods. 2025. doi:10.1017/rsm.2025.10031
BibTeX
@article{EmilovaDoneva2025,
author = {Emilova Doneva, Simona and de Viragh, Shirin and Hubarava, Hanna and Schandelmaier, Stefan and Briel, Matthias and Ineichen, Benjamin Victor},
title = {StudyTypeTeller—Large language models to automatically classify research study types for systematic reviews},
journal = {Research Synthesis Methods},
year = {2025},
doi = {10.1017/rsm.2025.10031}
}
EUVIMED