EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An evaluation of GPT models for phenotype concept recognition

Tudor Groza, Harry Caufield, Dylan Gration, Gareth Baynam, Melissa A Haendel, Peter N Robinson, Christopher J Mungall, Justin T Reese

BMC Medical Informatics and Decision Making · 2024

DOI: 10.1186/s12911-024-02439-w ↗

Kurz gefasst

Fachartikel in der Zeitschrift BMC Medical Informatics and Decision Making, 2024. Verfasst von Tudor Groza und 7 weiteren Autor:innen. Beteiligte Einrichtung: Rare Care Centre, Perth Children's Hospital, 15 Hospital Avenue, Nedlands, WA, 6009, Australia. tudor.groza@health.wa.gov.au. Hauptthemen laut MEDLINE: Knowledge, Language. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NHGRI NIH HHS, NIH HHS, National Institutes of Health.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Clinical deep phenotyping and phenotype annotation play a critical role in both the diagnosis of patients with rare disorders as well as in building computationally-tractable knowledge in the rare disorders field. These processes rely on using ontology concepts, often from the Human Phenotype Ontology, in conjunction with a phenotype concept recognition task (supported usually by machine learning methods) to curate patient profiles or existing scientific literature. With the significant shift in the use of large language models (LLMs) for most NLP tasks, we examine the performance of the latest Generative Pre-trained Transformer (GPT) models underpinning ChatGPT as a foundation for the tasks of clinical phenotyping and phenotype annotation. Materials and methods The experimental setup of the study included seven prompts of various levels of specificity, two GPT models (gpt-3.5-turbo and gpt-4.0) and two established gold standard corpora for phenotype recognition, one consisting of publication abstracts and the other clinical observations.

Results

The best run, using in-context learning, achieved 0.58 document-level F1 score on publication abstracts and 0.75 document-level F1 score on clinical observations, as well as a mention-level F1 score of 0.7, which surpasses the current best in class tool. Without in-context learning, however, performance is significantly below the existing approaches.

Conclusion

Our experiments show that gpt-4.0 surpasses the state of the art performance if the task is constrained to a subset of the target ontology where there is prior knowledge of the terms that are expected to be matched. While the results are promising, the non-deterministic nature of the outcomes, the high cost and the lack of concordance between different runs using the same prompt and input make the use of these LLMs challenging for this particular task.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

KnowledgeLanguageHumansMachine LearningPhenotypeRare Diseases

Stichwörter der Autor:innen

Artificial intelligenceGenerative pretrained transformerHuman Phenotype OntologyLarge language modelsPhenotype concept recognition

Autor:innen und Institutionen

  • Rare Care Centre, Perth Children's Hospital, 15 Hospital Avenue, Nedlands, WA, 6009, Australia. tudor.groza@health.wa.gov.au.
  • Telethon Kids Institute, 15 Hospital Avenue, Nedlands, WA, 6009, Australia. tudor.groza@health.wa.gov.au.
  • School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University, Kent St, Bentley, WA, 6102, Australia. tudor.groza@health.wa.gov.au.
  • SingHealth Duke-NUS Institute of Precision Medicine, 5 Hospital Drive Level 9, Singapore, 169609, Singapore. tudor.groza@health.wa.gov.au.
  • Division of Environmental Genomics and Systems Biology, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA.
  • Western Australian Register of Developmental Anomalies, King Edward Memorial Hospital, 374 Bagot Road, Subiaco, WA, 6008, Australia.

Förderung

  • NHGRI NIH HHS
  • NIH HHS
  • National Institutes of Health
  • The Director, Office of Science, Office of Basic Energy Sciences, of the US Department of Energy
  • Angela Wright Bennett Foundation
  • The Stan Perron Charitable Foundation, Australia
  • The McCusker Charitable Foundation via Channel 7 Telethon Trust
  • Mineral Resources via the Perth Children's Hospital Foundation

Zitieren

APA

Groza, T., Caufield, H., Gration, D., Baynam, G., Haendel, M. A., Robinson, P. N., Mungall, C. J., & Reese, J. T. (2024). An evaluation of GPT models for phenotype concept recognition. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-024-02439-w

Vancouver

Groza T, Caufield H, Gration D, Baynam G, Haendel MA, Robinson PN, et al. An evaluation of GPT models for phenotype concept recognition. BMC Medical Informatics and Decision Making. 2024. doi:10.1186/s12911-024-02439-w

BibTeX
@article{Groza2024,
  author = {Groza, Tudor and Caufield, Harry and Gration, Dylan and Baynam, Gareth and Haendel, Melissa A and Robinson, Peter N and Mungall, Christopher J and Reese, Justin T},
  title = {An evaluation of GPT models for phenotype concept recognition},
  journal = {BMC Medical Informatics and Decision Making},
  year = {2024},
  doi = {10.1186/s12911-024-02439-w}
}
Kostenlos starten →