Fine-tuning large language models for rare disease concept normalization
Journal of the American Medical Informatics Association · 2024
DOI: 10.1093/jamia/ocae133 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2024. Verfasst von Andy Wang und 3 weiteren Autor:innen. Beteiligte Einrichtung: Peddie School, Hightstown, NJ 08520, United States. Hauptthemen laut MEDLINE: Rare Diseases, Natural Language Processing, Biological Ontologies, Vocabulary, Controlled. Die Arbeit wurde bisher 47-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NHGRI NIH HHS, NIH HHS, NLM NIH HHS.
Abstract (Originalsprache)
Objective
We aim to develop a novel method for rare disease concept normalization by fine-tuning Llama 2, an open-source large language model (LLM), using a domain-specific corpus sourced from the Human Phenotype Ontology (HPO).
Methods
We developed an in-house template-based script to generate two corpora for fine-tuning. The first (NAME) contains standardized HPO names, sourced from the HPO vocabularies, along with their corresponding identifiers. The second (NAME+SYN) includes HPO names and half of the concept’s synonyms as well as identifiers. Subsequently, we fine-tuned Llama 2 (Llama2-7B) for each sentence set and conducted an evaluation using a range of sentence prompts and various phenotype terms.
Results
When the phenotype terms for normalization were included in the fine-tuning corpora, both models demonstrated nearly perfect performance, averaging over 99% accuracy. In comparison, ChatGPT-3.5 has only ∼20% accuracy in identifying HPO IDs for phenotype terms. When single-character typos were introduced in the phenotype terms, the accuracy of NAME and NAME+SYN is 10.2% and 36.1%, respectively, but increases to 61.8% (NAME+SYN) with additional typo-specific fine-tuning. For terms sourced from HPO vocabularies as unseen synonyms, the NAME model achieved 11.2% accuracy, while the NAME+SYN model achieved 92.7% accuracy.
Conclusion
Our fine-tuned models demonstrate ability to normalize phenotype terms unseen in the fine-tuning corpus, including misspellings, synonyms, terms from other ontologies, and laymen’s terms. Our approach provides a solution for the use of LLMs to identify named medical entities from clinical narratives, while successfully normalizing them to standard concepts in a controlled vocabulary.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Peddie School, Hightstown, NJ 08520, United States.
- Department of Biomedical Informatics, Columbia University, New York, NY 10032, United States.
- Department of Mathematics, University of Pennsylvania, Philadelphia, PA 19104, United States.
ORCID-Profile: Andy Wang, Cong Liu, Chunhua Weng
Förderung
- NHGRI NIH HHS
- NIH HHS
- NLM NIH HHS
- NIH
Zitieren
APA
Wang, A., Liu, C., Yang, J., & Weng, C. (2024). Fine-tuning large language models for rare disease concept normalization. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae133
Vancouver
Wang A, Liu C, Yang J, Weng C. Fine-tuning large language models for rare disease concept normalization. Journal of the American Medical Informatics Association. 2024. doi:10.1093/jamia/ocae133
BibTeX
@article{Wang2024,
author = {Wang, Andy and Liu, Cong and Yang, Jingye and Weng, Chunhua},
title = {Fine-tuning large language models for rare disease concept normalization},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
doi = {10.1093/jamia/ocae133}
}
EUVIMED