CACER: Clinical concept Annotations for Cancer Events and Relations
Journal of the American Medical Informatics Association · 2024
DOI: 10.1093/jamia/ocae231 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2024. Verfasst von Yujuan Velvin Fu und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics & Medical Education, University of Washington, Seattle, WA 98195, United States. Hauptthemen laut MEDLINE: Neoplasms, Natural Language Processing, Electronic Health Records. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NLM NIH HHS, NIH HHS, NIH.
Abstract (Originalsprache)
Objective
Clinical notes contain unstructured representations of patient histories, including the relationships between medical problems and prescription drugs. To investigate the relationship between cancer drugs and their associated symptom burden, we extract structured, semantic representations of medical problem and drug information from the clinical narratives of oncology notes. Materials and Methods We present Clinical concept Annotations for Cancer Events and Relations (CACER), a novel corpus with fine-grained annotations for over 48 000 medical problems and drug events and 10 000 drug-problem and problem-problem relations. Leveraging CACER, we develop and evaluate transformer-based information extraction models such as Bidirectional Encoder Representations from Transformers (BERT), Fine-tuned Language Net Text-To-Text Transfer Transformer (Flan-T5), Large Language Model Meta AI (Llama3), and Generative Pre-trained Transformers-4 (GPT-4) using fine-tuning and in-context learning (ICL).
Results
In event extraction, the fine-tuned BERT and Llama3 models achieved the highest performance at 88.2-88.0 F1, which is comparable to the inter-annotator agreement (IAA) of 88.4 F1. In relation extraction, the fine-tuned BERT, Flan-T5, and Llama3 achieved the highest performance at 61.8-65.3 F1. GPT-4 with ICL achieved the worst performance across both tasks.
Discussion
The fine-tuned models significantly outperformed GPT-4 in ICL, highlighting the importance of annotated training data and model optimization. Furthermore, the BERT models performed similarly to Llama3. For our task, large language models offer no performance advantage over the smaller BERT models.
Conclusions
We introduce CACER, a novel corpus with fine-grained annotations for medical problems, drugs, and their relationships in clinical narratives of oncology notes. State-of-the-art transformer models achieved performance comparable to IAA for several extraction tasks.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Substanzen
Autor:innen und Institutionen
- Department of Biomedical Informatics & Medical Education, University of Washington, Seattle, WA 98195, United States.
- Department of Information Sciences and Technology, George Mason University, Fairfax, VA 22030, United States.
- Huntsman Cancer Institute, University of Utah, Salt Lake City, UT 84112, United States.
- Department of Computer Science, Virginia Commonwealth University, Richmond, VA 23284, United States.
- Department of Linguistics, University of Washington, Seattle, WA 98195, United States.
ORCID-Profile: Giridhar Kaushik Ramachandran, Kevin Lybarger, Özlem Uzuner
Förderung
- NLM NIH HHS
- NIH HHS
- NIH
- NCI NIH HHS
- National Institutes of Health
- National Cancer Institute
- National Library of Medicine
Zitieren
APA
Fu, Y. V., Ramachandran, G. K., Halwani, A., McInnes, B. T., Xia, F., Lybarger, K., Yetisgen, M., & Uzuner, Ö. (2024). CACER: Clinical concept Annotations for Cancer Events and Relations. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae231
Vancouver
Fu YV, Ramachandran GK, Halwani A, McInnes BT, Xia F, Lybarger K, et al. CACER: Clinical concept Annotations for Cancer Events and Relations. Journal of the American Medical Informatics Association. 2024. doi:10.1093/jamia/ocae231
BibTeX
@article{Fu2024,
author = {Fu, Yujuan Velvin and Ramachandran, Giridhar Kaushik and Halwani, Ahmad and McInnes, Bridget T and Xia, Fei and Lybarger, Kevin and Yetisgen, Meliha and Uzuner, Özlem},
title = {CACER: Clinical concept Annotations for Cancer Events and Relations},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
doi = {10.1093/jamia/ocae231}
}
EUVIMED