Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy
Information Research an international electronic journal · 20.03.2026 · 31 (iConf) · 1502-1514
DOI: 10.47989/ir31iconf64265 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift Information Research an international electronic journal, online erschienen am 20.03.2026. Verfasst von Wonchan Choi und 4 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Introduction
Literature screening is among the most time- and labour-intensive phases of systematic literature reviews (SLRs). Although large language models (LLMs) have been explored for screening, prior work has focused on medical and environmental domains and mainly benchmarked LLMs against human coders, offering limited guidance on collaborative integration in SLRs.
Method
This study evaluated 12 GPT model–prompt configurations in an SLR of 1,616 publications. Two human coders screened a 10% sample (n = 162) to create a gold standard for model comparison. Performance was assessed using balanced accuracy, recall for inclusion, time efficiency, and cost efficiency. Disagreements between humans and models were analysed. Analysis. Bootstrap tests compared performance across configurations. Open coding identified error types in human–model disagreements. Group discussion resolved discrepancies.
Results
GPT-5 Zeroshot and GPT-4o-mini Fewshot achieved the highest performance (accuracy = 0.990 and 0.946; recall = 1.000 and 0.933). GPT-4o-mini was faster and cheaper but more prone to overly rigid rule application. Error analysis identified 10 mismatches, leading to two corrections of human miscoding.
Conclusion
LLM-assisted screening can reduce workload, improve efficiency, and correct human errors in SLR. Practical guidelines for prompt design and confidence thresholds can position LLMs as collaborative tools in SLRs.
Zitieren
APA
Choi, W., Lee, J., Stvilia, B., Zhang, Y., & Bak, H. (2026). Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy. Information Research an international electronic journal, 31(iConf), 1502-1514. https://doi.org/10.47989/ir31iconf64265
Vancouver
Choi W, Lee J, Stvilia B, Zhang Y, Bak H. Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy. Information Research an international electronic journal. 2026;31(iConf):1502-1514. doi:10.47989/ir31iconf64265
BibTeX
@article{Choi2026,
author = {Choi, Wonchan and Lee, Joyce and Stvilia, Besiki and Zhang, Yan and Bak, Hyerin},
title = {Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy},
journal = {Information Research an international electronic journal},
year = {2026},
volume = {31},
number = {iConf},
pages = {1502--1514},
doi = {10.47989/ir31iconf64265}
}
EUVIMED