EUVIMEDEUVIMED entdecken
Systematische ÜbersichtOpen Access · CC BY-NC 4.0

Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy

Wonchan Choi, Joyce Lee, Besiki Stvilia, Yan Zhang, Hyerin Bak

Information Research an international electronic journal · 20.03.2026 · 31 (iConf) · 1502-1514

DOI: 10.47989/ir31iconf64265 ↗

Kurz gefasst

Systematische Übersicht in der Zeitschrift Information Research an international electronic journal, online erschienen am 20.03.2026. Verfasst von Wonchan Choi und 4 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Literature screening is among the most time- and labour-intensive phases of systematic literature reviews (SLRs). Although large language models (LLMs) have been explored for screening, prior work has focused on medical and environmental domains and mainly benchmarked LLMs against human coders, offering limited guidance on collaborative integration in SLRs.

Method

This study evaluated 12 GPT model–prompt configurations in an SLR of 1,616 publications. Two human coders screened a 10% sample (n = 162) to create a gold standard for model comparison. Performance was assessed using balanced accuracy, recall for inclusion, time efficiency, and cost efficiency. Disagreements between humans and models were analysed. Analysis. Bootstrap tests compared performance across configurations. Open coding identified error types in human–model disagreements. Group discussion resolved discrepancies.

Results

GPT-5 Zeroshot and GPT-4o-mini Fewshot achieved the highest performance (accuracy = 0.990 and 0.946; recall = 1.000 and 0.933). GPT-4o-mini was faster and cheaper but more prone to overly rigid rule application. Error analysis identified 10 mismatches, leading to two corrections of human miscoding.

Conclusion

LLM-assisted screening can reduce workload, improve efficiency, and correct human errors in SLR. Practical guidelines for prompt design and confidence thresholds can position LLMs as collaborative tools in SLRs.

Zitieren

APA

Choi, W., Lee, J., Stvilia, B., Zhang, Y., & Bak, H. (2026). Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy. Information Research an international electronic journal, 31(iConf), 1502-1514. https://doi.org/10.47989/ir31iconf64265

Vancouver

Choi W, Lee J, Stvilia B, Zhang Y, Bak H. Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy. Information Research an international electronic journal. 2026;31(iConf):1502-1514. doi:10.47989/ir31iconf64265

BibTeX
@article{Choi2026,
  author = {Choi, Wonchan and Lee, Joyce and Stvilia, Besiki and Zhang, Yan and Bak, Hyerin},
  title = {Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy},
  journal = {Information Research an international electronic journal},
  year = {2026},
  volume = {31},
  number = {iConf},
  pages = {1502--1514},
  doi = {10.47989/ir31iconf64265}
}
Kostenlos starten →