EUVIMEDEUVIMED entdecken
Crossref

Performance of two large language models for data extraction in evidence synthesis

Amanda Konet iD, Ian Thomas, Gerald Gartlehner iD, Leila Kahwati iD, Rainer Hilscher, Shannon Kugley, Karen Crotty, Meera Viswanathan, Robert Chew

Research Synthesis Methods · 2024-09 · 15 (5) · 818-824

DOI: 10.1002/jrsm.1732 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research Synthesis Methods, online erschienen am 19.06.2024. Verfasst von Amanda Konet und 8 weiteren Autor:innen. Beteiligte Einrichtung: Social, Statistical, and Environmental Sciences, RTI International, Research Triangle Park, North Carolina, USA. Hauptthemen laut MEDLINE: Software. Die Arbeit wurde bisher 87-mal zitiert (Crossref) und nennt 28 Literaturangaben. Gefördert von: RTI International.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT‐4) for extracting pre‐specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT‐4. GPT‐4 required use of a third‐party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT‐4 with the plug‐in was lower (68.8%); however, most of the errors were due to the plug‐in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT‐4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open‐source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

SoftwareHumansReproducibility of ResultsInformation Storage and RetrievalNatural Language ProcessingAlgorithmsLanguageSystematic Reviews as TopicProgramming LanguagesResearch Design

Stichwörter der Autor:innen

accuracyartificial intelligencedata extractionevidence synthesislarge language models

Autor:innen und Institutionen

  • Social, Statistical, and Environmental Sciences, RTI International, Research Triangle Park, North Carolina, USA.
  • Department for Evidence-based Medicine and Evaluation, Danube University Krems, Krems, Austria.

ORCID-Profile: Amanda Konet, Gerald Gartlehner, Leila Kahwati

Förderung

  • RTI International

Zitieren

APA

Konet, A., Thomas, I., Gartlehner, G., Kahwati, L., Hilscher, R., Kugley, S., Crotty, K., Viswanathan, M., & Chew, R. (2024). Performance of two large language models for data extraction in evidence synthesis. Research Synthesis Methods, 15(5), 818-824. https://doi.org/10.1002/jrsm.1732

Vancouver

Konet A, Thomas I, Gartlehner G, Kahwati L, Hilscher R, Kugley S, et al. Performance of two large language models for data extraction in evidence synthesis. Research Synthesis Methods. 2024;15(5):818-824. doi:10.1002/jrsm.1732

BibTeX
@article{Konet2024,
  author = {Konet, Amanda and Thomas, Ian and Gartlehner, Gerald and Kahwati, Leila and Hilscher, Rainer and Kugley, Shannon and Crotty, Karen and Viswanathan, Meera and Chew, Robert},
  title = {Performance of two large language models for data extraction in evidence synthesis},
  journal = {Research Synthesis Methods},
  year = {2024},
  volume = {15},
  number = {5},
  pages = {818--824},
  doi = {10.1002/jrsm.1732}
}
Kostenlos starten →