Distilling large language models for matching patients to clinical trials
Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1953-1963
DOI: 10.1093/jamia/ocae073 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 19.04.2024. Verfasst von Mauro Nievas und 3 weiteren Autor:innen. Beteiligte Einrichtung: Triomics Research, Triomics, Inc., San Francisco, CA 94105, United States. Hauptthemen laut MEDLINE: Clinical Trials as Topic, Patient Selection. Die Arbeit wurde bisher 71-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Objective
The objective of this study is to systematically examine the efficacy of both proprietary (GPT-3.5, GPT-4) and open-source large language models (LLMs) (LLAMA 7B, 13B, 70B) in the context of matching patients to clinical trials in healthcare. Materials and methods The study employs a multifaceted evaluation framework, incorporating extensive automated and human-centric assessments along with a detailed error analysis for each model, and assesses LLMs’ capabilities in analyzing patient eligibility against clinical trial’s inclusion and exclusion criteria. To improve the adaptability of open-source LLMs, a specialized synthetic dataset was created using GPT-4, facilitating effective fine-tuning under constrained data conditions.
Results
The findings indicate that open-source LLMs, when fine-tuned on this limited and synthetic dataset, achieve performance parity with their proprietary counterparts, such as GPT-3.5.
Discussion
This study highlights the recent success of LLMs in the high-stakes domain of healthcare, specifically in patient-trial matching. The research demonstrates the potential of open-source models to match the performance of proprietary models when fine-tuned appropriately, addressing challenges like cost, privacy, and reproducibility concerns associated with closed-source proprietary LLMs.
Conclusion
The study underscores the opportunity for open-source LLMs in patient-trial matching. To encourage further research and applications in this field, the annotated evaluation dataset and the fine-tuned LLM, Trial-LLAMA, are released for public use.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Triomics Research, Triomics, Inc., San Francisco, CA 94105, United States.
- Triomics Research, Triomics, Inc., Bengaluru, Karnataka 560102, India.
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15260, United States.
Zitieren
APA
Nievas, M., Basu, A., Wang, Y., & Singh, H. (2024). Distilling large language models for matching patients to clinical trials. Journal of the American Medical Informatics Association, 31(9), 1953-1963. https://doi.org/10.1093/jamia/ocae073
Vancouver
Nievas M, Basu A, Wang Y, Singh H. Distilling large language models for matching patients to clinical trials. Journal of the American Medical Informatics Association. 2024;31(9):1953-1963. doi:10.1093/jamia/ocae073
BibTeX
@article{Nievas2024,
author = {Nievas, Mauro and Basu, Aditya and Wang, Yanshan and Singh, Hrituraj},
title = {Distilling large language models for matching patients to clinical trials},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {9},
pages = {1953--1963},
doi = {10.1093/jamia/ocae073}
}
EUVIMED