A Hybrid Delphi-Inspired Expert–LLM Workflow for Efficient Evidence Screening in Systematic Reviews
Studies in Health Technology and Informatics · 21.05.2026
DOI: 10.3233/shti260255 ↗Kurz gefasst
Delphi-Studie in der Zeitschrift Studies in Health Technology and Informatics, erschienen am 21.05.2026. Verfasst von Omid Pournik und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electronic, Electrical and Systems Engineering, University of Birmingham, UK. Hauptthemen laut MEDLINE: Workflow, Delphi Technique, Systematic Reviews as Topic, Thyroid Nodule. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Systematic reviews are essential for evidence-based healthcare but remain highly resource-intensive, with most retrieved studies ultimately excluded after manual screening. This study developed and evaluated a hybrid expert–LLM workflow to reduce human workload while maintaining accuracy and transparency. Within the Thyroid Risk Stratification Tool (ThyRST) project, ChatGPT-5 was used to classify 14,858 records on thyroid nodule malignancy risk into thematic categories. Recurrent irrelevant concepts were refined through expert consensus involving clinicians and informaticians and embedded as exclusion rules in structured prompts. The model then labelled each abstract as INCLUDE, EXCLUDE, or MAYBE, producing outputs for audit and verification. A random sample of 100 records was independently reviewed by human assessors to evaluate performance. The workflow achieved 96% concordance (κ = 0.91) with human reviewers, with only one false exclusion, and reduced manual screening time by approximately 70%. These results demonstrate that a transparent Delphi-inspired expert–LLM can accurately and reproducibly automate early-stage evidence screening, providing substantial efficiency gains while preserving human oversight and methodological rigor. The approach offers a practical pathway toward the responsible integration of generative AI in systematic review methodology and digital health research.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Electronic, Electrical and Systems Engineering, University of Birmingham, UK.
- Department of Applied Health Sciences, University of Birmingham, UK.
- Department of Otolaryngology and Head and Neck Surgery, University Hospitals Birmingham, UK.
- Department of Endocrinology, University Hospitals Birmingham.
Zitieren
APA
Pournik, O., Watts, E., Richards, E., Boelaert, K., Sharma, N., Abbasi, S. F., Ding, X., Ghalichi, L., & Arvanitis, T. N. (2026). A Hybrid Delphi-Inspired Expert–LLM Workflow for Efficient Evidence Screening in Systematic Reviews. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti260255
Vancouver
Pournik O, Watts E, Richards E, Boelaert K, Sharma N, Abbasi SF, et al. A Hybrid Delphi-Inspired Expert–LLM Workflow for Efficient Evidence Screening in Systematic Reviews. Studies in Health Technology and Informatics. 2026. doi:10.3233/shti260255
BibTeX
@article{Pournik2026,
author = {Pournik, Omid and Watts, Emma and Richards, Emma and Boelaert, Kristien and Sharma, Neil and Abbasi, Saadullah Farooq and Ding, Xuefei and Ghalichi, Leila and Arvanitis, Theodoros N},
title = {A Hybrid Delphi-Inspired Expert–LLM Workflow for Efficient Evidence Screening in Systematic Reviews},
journal = {Studies in Health Technology and Informatics},
year = {2026},
doi = {10.3233/shti260255}
}
EUVIMED