Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score
JCO Clinical Cancer Informatics · 2024
DOI: 10.1200/cci.24.00123 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JCO Clinical Cancer Informatics, 2024. Verfasst von Zacharie Hamilton und 10 weiteren Autor:innen. Beteiligte Einrichtung: University of Illinois Chicago, Chicago, IL. Hauptthemen laut MEDLINE: Carcinoma, Non-Small-Cell Lung, Lung Neoplasms, Oncogenes, Artificial Intelligence. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
PURPOSE Precision oncology in non–small cell lung cancer (NSCLC) relies on biomarker testing for clinical decision making. Despite its importance, challenges like the lack of genomic oncology training, nonstandardized biomarker reporting, and a rapidly evolving treatment landscape hinder its practice. Generative artificial intelligence (AI), such as ChatGPT, offers promise for enhancing clinical decision support. Effective performance metrics are crucial to evaluate these models' accuracy and their propensity for producing incorrect or hallucinated information. We assessed various ChatGPT versions' ability to generate accurate next-generation sequencing reports and treatment recommendations for NSCLC, using a novel Generative AI Performance Score (G-PS), which considers accuracy, relevancy, and hallucinations. METHODS We queried ChatGPT versions for first-line NSCLC treatment recommendations with an Food and Drug Administration–approved targeted therapy, using a zero-shot prompt approach for eight oncogenes. Responses were assessed against National Comprehensive Cancer Network (NCCN) guidelines for accuracy, relevance, and hallucinations, with G-PS calculating scores from –1 (all hallucinations) to 1 (fully NCCN-compliant recommendations). G-PS was designed as a composite measure with a base score for correct recommendations (weighted for preferred treatments) and a penalty for hallucinations. RESULTS Analyzing 160 responses, generative pre-trained transformer (GPT)-4 outperformed GPT-3.5, showing higher base score (90% v 60%; P < .01) and fewer hallucinations (34% v 53%; P < .01). GPT-4's overall G-PS was significantly higher (0.34 v –0.15; P < .01), indicating superior performance. CONCLUSION This study highlights the rapid improvement of generative AI in matching treatment recommendations with biomarkers in precision oncology. Although the rate of hallucinations improved in the GPT-4 model, future generative AI use in clinical care requires high levels of accuracy with minimal to no room for hallucinations. The GP-S represents a novel metric quantifying generative AI utility in health care compared with national guidelines, with potential adaptation beyond precision oncology.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Substanzen
Autor:innen und Institutionen
- University of Illinois Chicago, Chicago, IL.
- University of Washington, Seattle, WA.
- Kirkland & Ellis, Chicago, IL.
ORCID-Profile: Zacharie Hamilton, Aseem Aseem, Zhengjia Chen, Noor Naffakh, Natalie M Reizine, Frank Weinberg, Shikha Jain, Vijayakrishna K Gadi, Ryan H Nguyen
Zitieren
APA
Hamilton, Z., Aseem, A., Chen, Z., Naffakh, N., Reizine, N. M., Weinberg, F., Jain, S., Kessler, L. G., Gadi, V. K., Bun, C., & Nguyen, R. H. (2024). Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score. JCO Clinical Cancer Informatics. https://doi.org/10.1200/cci.24.00123
Vancouver
Hamilton Z, Aseem A, Chen Z, Naffakh N, Reizine NM, Weinberg F, et al. Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score. JCO Clinical Cancer Informatics. 2024. doi:10.1200/cci.24.00123
BibTeX
@article{Hamilton2024,
author = {Hamilton, Zacharie and Aseem, Aseem and Chen, Zhengjia and Naffakh, Noor and Reizine, Natalie M and Weinberg, Frank and Jain, Shikha and Kessler, Larry G and Gadi, Vijayakrishna K and Bun, Christopher and Nguyen, Ryan H},
title = {Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score},
journal = {JCO Clinical Cancer Informatics},
year = {2024},
doi = {10.1200/cci.24.00123}
}
EUVIMED