AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models
Journal of the American Medical Informatics Association · 18.01.2024 · 31 (2) · 375-385
DOI: 10.1093/jamia/ocad218 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 11.11.2023. Verfasst von Surabhi Datta und 11 weiteren Autor:innen. Beteiligte Einrichtung: Melax Technologies, Houston, TX 77030, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Breast Neoplasms. Die Arbeit wurde bisher 54-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Objectives
We aim to build a generalizable information extraction system leveraging large language models to extract granular eligibility criteria information for diverse diseases from free text clinical trial protocol documents. We investigate the model’s capability to extract criteria entities along with contextual attributes including values, temporality, and modifiers and present the strengths and limitations of this system. Materials and Methods The clinical trial data were acquired from https://ClinicalTrials.gov/. We developed a system, AutoCriteria, which comprises the following modules: preprocessing, knowledge ingestion, prompt modeling based on GPT, postprocessing, and interim evaluation. The final system evaluation was performed, both quantitatively and qualitatively, on 180 manually annotated trials encompassing 9 diseases.
Results
AutoCriteria achieves an overall F1 score of 89.42 across all 9 diseases in extracting the criteria entities, with the highest being 95.44 for nonalcoholic steatohepatitis and the lowest of 84.10 for breast cancer. Its overall accuracy is 78.95% in identifying all contextual information across all diseases. Our thematic analysis indicated accurate logic interpretation of criteria as one of the strengths and overlooking/neglecting the main criteria as one of the weaknesses of AutoCriteria.
Discussion
AutoCriteria demonstrates strong potential to extract granular eligibility criteria information from trial documents without requiring manual annotations. The prompts developed for AutoCriteria generalize well across different disease areas. Our evaluation suggests that the system handles complex scenarios including multiple arm conditions and logics.
Conclusion
AutoCriteria currently encompasses a diverse range of diseases and holds potential to extend to more in the future. This signifies a generalizable and scalable solution, poised to address the complexities of clinical trial application in real-world settings.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Melax Technologies, Houston, TX 77030, United States.
- Regeneron Pharmaceuticals, Tarrytown, NY 10591, United States.
- Yale School of Medicine, New Haven, CT 06511, United States.
ORCID-Profile: Surabhi Datta, Jingcheng Du
Zitieren
APA
Datta, S., Lee, K., Paek, H., Manion, F. J., Ofoegbu, N., Du, J., Li, Y., Huang, L. C., Wang, J., Lin, B., Xu, H., & Wang, X. (2024). AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models. Journal of the American Medical Informatics Association, 31(2), 375-385. https://doi.org/10.1093/jamia/ocad218
Vancouver
Datta S, Lee K, Paek H, Manion FJ, Ofoegbu N, Du J, et al. AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models. Journal of the American Medical Informatics Association. 2024;31(2):375-385. doi:10.1093/jamia/ocad218
BibTeX
@article{Datta2024,
author = {Datta, Surabhi and Lee, Kyeryoung and Paek, Hunki and Manion, Frank J and Ofoegbu, Nneka and Du, Jingcheng and Li, Ying and Huang, Liang-Chin and Wang, Jingqi and Lin, Bin and Xu, Hua and Wang, Xiaoyan},
title = {AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {2},
pages = {375--385},
doi = {10.1093/jamia/ocad218}
}
EUVIMED