EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models

Surabhi Datta iD, Kyeryoung Lee, Hunki Paek, Frank J Manion, Nneka Ofoegbu, Jingcheng Du iD, Ying Li, Liang-Chin Huang, Jingqi Wang, Bin Lin, Hua Xu, Xiaoyan Wang

Journal of the American Medical Informatics Association · 18.01.2024 · 31 (2) · 375-385

DOI: 10.1093/jamia/ocad218 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 11.11.2023. Verfasst von Surabhi Datta und 11 weiteren Autor:innen. Beteiligte Einrichtung: Melax Technologies, Houston, TX 77030, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Breast Neoplasms. Die Arbeit wurde bisher 54-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

We aim to build a generalizable information extraction system leveraging large language models to extract granular eligibility criteria information for diverse diseases from free text clinical trial protocol documents. We investigate the model’s capability to extract criteria entities along with contextual attributes including values, temporality, and modifiers and present the strengths and limitations of this system. Materials and Methods The clinical trial data were acquired from https://ClinicalTrials.gov/. We developed a system, AutoCriteria, which comprises the following modules: preprocessing, knowledge ingestion, prompt modeling based on GPT, postprocessing, and interim evaluation. The final system evaluation was performed, both quantitatively and qualitatively, on 180 manually annotated trials encompassing 9 diseases.

Results

AutoCriteria achieves an overall F1 score of 89.42 across all 9 diseases in extracting the criteria entities, with the highest being 95.44 for nonalcoholic steatohepatitis and the lowest of 84.10 for breast cancer. Its overall accuracy is 78.95% in identifying all contextual information across all diseases. Our thematic analysis indicated accurate logic interpretation of criteria as one of the strengths and overlooking/neglecting the main criteria as one of the weaknesses of AutoCriteria.

Discussion

AutoCriteria demonstrates strong potential to extract granular eligibility criteria information from trial documents without requiring manual annotations. The prompts developed for AutoCriteria generalize well across different disease areas. Our evaluation suggests that the system handles complex scenarios including multiple arm conditions and logics.

Conclusion

AutoCriteria currently encompasses a diverse range of diseases and holds potential to extend to more in the future. This signifies a generalizable and scalable solution, poised to address the complexities of clinical trial application in real-world settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingBreast NeoplasmsHumansFemaleInformation Storage and RetrievalLanguageEligibility Determination

Stichwörter der Autor:innen

GPT-4clinical trial eligibility criteriainformation extractionlarge language modelnatural language processingzero-shot prompting

Autor:innen und Institutionen

  • Melax Technologies, Houston, TX 77030, United States.
  • Regeneron Pharmaceuticals, Tarrytown, NY 10591, United States.
  • Yale School of Medicine, New Haven, CT 06511, United States.

ORCID-Profile: Surabhi Datta, Jingcheng Du

Zitieren

APA

Datta, S., Lee, K., Paek, H., Manion, F. J., Ofoegbu, N., Du, J., Li, Y., Huang, L. C., Wang, J., Lin, B., Xu, H., & Wang, X. (2024). AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models. Journal of the American Medical Informatics Association, 31(2), 375-385. https://doi.org/10.1093/jamia/ocad218

Vancouver

Datta S, Lee K, Paek H, Manion FJ, Ofoegbu N, Du J, et al. AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models. Journal of the American Medical Informatics Association. 2024;31(2):375-385. doi:10.1093/jamia/ocad218

BibTeX
@article{Datta2024,
  author = {Datta, Surabhi and Lee, Kyeryoung and Paek, Hunki and Manion, Frank J and Ofoegbu, Nneka and Du, Jingcheng and Li, Ying and Huang, Liang-Chin and Wang, Jingqi and Lin, Bin and Xu, Hua and Wang, Xiaoyan},
  title = {AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {2},
  pages = {375--385},
  doi = {10.1093/jamia/ocad218}
}
Kostenlos starten →