EUVIMEDEUVIMED entdecken
Open Access

Fine-tuning large language models for rare disease concept normalization

Andy Wang iD, Cong Liu iD, Jingye Yang, Chunhua Weng iD

Journal of the American Medical Informatics Association · 2024

DOI: 10.1093/jamia/ocae133 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2024. Verfasst von Andy Wang und 3 weiteren Autor:innen. Beteiligte Einrichtung: Peddie School, Hightstown, NJ 08520, United States. Hauptthemen laut MEDLINE: Rare Diseases, Natural Language Processing, Biological Ontologies, Vocabulary, Controlled. Die Arbeit wurde bisher 47-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NHGRI NIH HHS, NIH HHS, NLM NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

We aim to develop a novel method for rare disease concept normalization by fine-tuning Llama 2, an open-source large language model (LLM), using a domain-specific corpus sourced from the Human Phenotype Ontology (HPO).

Methods

We developed an in-house template-based script to generate two corpora for fine-tuning. The first (NAME) contains standardized HPO names, sourced from the HPO vocabularies, along with their corresponding identifiers. The second (NAME+SYN) includes HPO names and half of the concept’s synonyms as well as identifiers. Subsequently, we fine-tuned Llama 2 (Llama2-7B) for each sentence set and conducted an evaluation using a range of sentence prompts and various phenotype terms.

Results

When the phenotype terms for normalization were included in the fine-tuning corpora, both models demonstrated nearly perfect performance, averaging over 99% accuracy. In comparison, ChatGPT-3.5 has only ∼20% accuracy in identifying HPO IDs for phenotype terms. When single-character typos were introduced in the phenotype terms, the accuracy of NAME and NAME+SYN is 10.2% and 36.1%, respectively, but increases to 61.8% (NAME+SYN) with additional typo-specific fine-tuning. For terms sourced from HPO vocabularies as unseen synonyms, the NAME model achieved 11.2% accuracy, while the NAME+SYN model achieved 92.7% accuracy.

Conclusion

Our fine-tuned models demonstrate ability to normalize phenotype terms unseen in the fine-tuning corpus, including misspellings, synonyms, terms from other ontologies, and laymen’s terms. Our approach provides a solution for the use of LLMs to identify named medical entities from clinical narratives, while successfully normalizing them to standard concepts in a controlled vocabulary.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Rare DiseasesNatural Language ProcessingBiological OntologiesVocabulary, ControlledPhenotypeHumans

Stichwörter der Autor:innen

HPOLlama 2concept normalizationfine-tuninglarge language model

Autor:innen und Institutionen

  • Peddie School, Hightstown, NJ 08520, United States.
  • Department of Biomedical Informatics, Columbia University, New York, NY 10032, United States.
  • Department of Mathematics, University of Pennsylvania, Philadelphia, PA 19104, United States.

ORCID-Profile: Andy Wang, Cong Liu, Chunhua Weng

Förderung

  • NHGRI NIH HHS
  • NIH HHS
  • NLM NIH HHS
  • NIH

Zitieren

APA

Wang, A., Liu, C., Yang, J., & Weng, C. (2024). Fine-tuning large language models for rare disease concept normalization. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae133

Vancouver

Wang A, Liu C, Yang J, Weng C. Fine-tuning large language models for rare disease concept normalization. Journal of the American Medical Informatics Association. 2024. doi:10.1093/jamia/ocae133

BibTeX
@article{Wang2024,
  author = {Wang, Andy and Liu, Cong and Yang, Jingye and Weng, Chunhua},
  title = {Fine-tuning large language models for rare disease concept normalization},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  doi = {10.1093/jamia/ocae133}
}
Kostenlos starten →