Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models
PharmacoEconomics - Open · 2024-03 · 8 (2) · 205-220
DOI: 10.1007/s41669-024-00476-9 ↗Kurz gefasst
Netzwerk-Metaanalyse in der Zeitschrift PharmacoEconomics - Open, online erschienen am 10.02.2024. Verfasst von Tim Reason und 5 weiteren Autor:innen. Beteiligte Einrichtung: Estima Scientific, Mediaworks, 191 Wood Lane, London, W12 7FP, UK. tim.reason@estima-sci.com. Die Arbeit wurde bisher 81-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Bristol-Myers Squibb.
Abstract (Originalsprache)
BACKGROUND: The emergence of artificial intelligence, capable of human-level performance on some tasks, presents an opportunity to revolutionise development of systematic reviews and network meta-analyses (NMAs). In this pilot study, we aim to assess use of a large-language model (LLM, Generative Pre-trained Transformer 4 [GPT-4]) to automatically extract data from publications, write an R script to conduct an NMA and interpret the results. METHODS: We considered four case studies involving binary and time-to-event outcomes in two disease areas, for which an NMA had previously been conducted manually. For each case study, a Python script was developed that communicated with the LLM via application programming interface (API) calls. The LLM was prompted to extract relevant data from publications, to create an R script to be used to run the NMA and then to produce a small report describing the analysis. RESULTS: The LLM had a > 99% success rate of accurately extracting data across 20 runs for each case study and could generate R scripts that could be run end-to-end without human input. It also produced good quality reports describing the disease area, analysis conducted, results obtained and a correct interpretation of the results. CONCLUSIONS: This study provides a promising indication of the feasibility of using current generation LLMs to automate data extraction, code generation and NMA result interpretation, which could result in significant time savings and reduce human error. This is provided that routine technical checks are performed, as recommend for human-conducted analyses. Whilst not currently 100% consistent, LLMs are likely to improve with time.
Autor:innen und Institutionen
- Estima Scientific, Mediaworks, 191 Wood Lane, London, W12 7FP, UK. tim.reason@estima-sci.com.
- Estima Scientific, Mediaworks, 191 Wood Lane, London, W12 7FP, UK.
- Bristol Myers Squibb, Princeton, NJ, USA.
- Bristol Myers Squibb, Uxbridge, UK.
Förderung
- Bristol-Myers Squibb
Zitieren
APA
Reason, T., Benbow, E., Langham, J., Gimblett, A., Klijn, S. L., & Malcolm, B. (2024). Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models. PharmacoEconomics - Open, 8(2), 205-220. https://doi.org/10.1007/s41669-024-00476-9
Vancouver
Reason T, Benbow E, Langham J, Gimblett A, Klijn SL, Malcolm B. Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models. PharmacoEconomics - Open. 2024;8(2):205-220. doi:10.1007/s41669-024-00476-9
BibTeX
@article{Reason2024,
author = {Reason, Tim and Benbow, Emma and Langham, Julia and Gimblett, Andy and Klijn, Sven L and Malcolm, Bill},
title = {Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models},
journal = {PharmacoEconomics - Open},
year = {2024},
volume = {8},
number = {2},
pages = {205--220},
doi = {10.1007/s41669-024-00476-9}
}
EUVIMED