EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Data extraction for evidence synthesis using a large language model: A proof‐of‐concept study

Gerald Gartlehner iD, Leila Kahwati iD, Rainer Hilscher, Ian Thomas, Shannon Kugley, Karen Crotty, Meera Viswanathan, Barbara Nussbaumer-Streit, Graham Booth, Nathaniel Erskine iD, Amanda Konet iD, Robert Chew

Research Synthesis Methods · 2024-07 · 15 (4) · 576-589

DOI: 10.1002/jrsm.1710 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research Synthesis Methods, online erschienen am 03.03.2024. Verfasst von Gerald Gartlehner und 11 weiteren Autor:innen. Beteiligte Einrichtung: Social, Statistical, and Environmental Sciences, RTI International, Research Triangle Park, North Carolina, USA. Hauptthemen laut MEDLINE: Proof of Concept Study, Machine Learning. Die Arbeit wurde bisher 190-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: RTI International Innovation Office.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Data extraction is a crucial, yet labor‐intensive and error‐prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof‐of‐concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English‐language, open‐access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test–retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors ( n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero‐shot learning). Based on findings of our proof‐of‐concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Proof of Concept StudyMachine LearningHumansReproducibility of ResultsSystematic Reviews as TopicRandomized Controlled Trials as TopicAlgorithmsInformation Storage and RetrievalLanguageSoftwareNatural Language ProcessingResearch Design

Stichwörter der Autor:innen

accuracyartificial intelligencedata extractionevidence synthesislarge language modelsproof of concept

Autor:innen und Institutionen

  • Social, Statistical, and Environmental Sciences, RTI International, Research Triangle Park, North Carolina, USA.
  • Department for Evidence-based Medicine and Evaluation, Danube University Krems, Krems, Austria.
  • Preventive Medicine Residency Program, Department of Family Medicine, School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA.

ORCID-Profile: Gerald Gartlehner, Leila Kahwati, Nathaniel Erskine, Amanda Konet

Förderung

  • RTI International Innovation Office

Zitieren

APA

Gartlehner, G., Kahwati, L., Hilscher, R., Thomas, I., Kugley, S., Crotty, K., Viswanathan, M., Nussbaumer-Streit, B., Booth, G., Erskine, N., Konet, A., & Chew, R. (2024). Data extraction for evidence synthesis using a large language model: A proof‐of‐concept study. Research Synthesis Methods, 15(4), 576-589. https://doi.org/10.1002/jrsm.1710

Vancouver

Gartlehner G, Kahwati L, Hilscher R, Thomas I, Kugley S, Crotty K, et al. Data extraction for evidence synthesis using a large language model: A proof‐of‐concept study. Research Synthesis Methods. 2024;15(4):576-589. doi:10.1002/jrsm.1710

BibTeX
@article{Gartlehner2024,
  author = {Gartlehner, Gerald and Kahwati, Leila and Hilscher, Rainer and Thomas, Ian and Kugley, Shannon and Crotty, Karen and Viswanathan, Meera and Nussbaumer-Streit, Barbara and Booth, Graham and Erskine, Nathaniel and Konet, Amanda and Chew, Robert},
  title = {Data extraction for evidence synthesis using a large language model: A proof‐of‐concept study},
  journal = {Research Synthesis Methods},
  year = {2024},
  volume = {15},
  number = {4},
  pages = {576--589},
  doi = {10.1002/jrsm.1710}
}
Kostenlos starten →