EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score

Zacharie Hamilton iD, Aseem Aseem iD, Zhengjia Chen iD, Noor Naffakh iD, Natalie M Reizine iD, Frank Weinberg iD, Shikha Jain iD, Larry G Kessler, Vijayakrishna K Gadi iD, Christopher Bun, Ryan H Nguyen iD

JCO Clinical Cancer Informatics · 2024

DOI: 10.1200/cci.24.00123 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JCO Clinical Cancer Informatics, 2024. Verfasst von Zacharie Hamilton und 10 weiteren Autor:innen. Beteiligte Einrichtung: University of Illinois Chicago, Chicago, IL. Hauptthemen laut MEDLINE: Carcinoma, Non-Small-Cell Lung, Lung Neoplasms, Oncogenes, Artificial Intelligence. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

PURPOSE Precision oncology in non–small cell lung cancer (NSCLC) relies on biomarker testing for clinical decision making. Despite its importance, challenges like the lack of genomic oncology training, nonstandardized biomarker reporting, and a rapidly evolving treatment landscape hinder its practice. Generative artificial intelligence (AI), such as ChatGPT, offers promise for enhancing clinical decision support. Effective performance metrics are crucial to evaluate these models' accuracy and their propensity for producing incorrect or hallucinated information. We assessed various ChatGPT versions' ability to generate accurate next-generation sequencing reports and treatment recommendations for NSCLC, using a novel Generative AI Performance Score (G-PS), which considers accuracy, relevancy, and hallucinations. METHODS We queried ChatGPT versions for first-line NSCLC treatment recommendations with an Food and Drug Administration–approved targeted therapy, using a zero-shot prompt approach for eight oncogenes. Responses were assessed against National Comprehensive Cancer Network (NCCN) guidelines for accuracy, relevance, and hallucinations, with G-PS calculating scores from –1 (all hallucinations) to 1 (fully NCCN-compliant recommendations). G-PS was designed as a composite measure with a base score for correct recommendations (weighted for preferred treatments) and a penalty for hallucinations. RESULTS Analyzing 160 responses, generative pre-trained transformer (GPT)-4 outperformed GPT-3.5, showing higher base score (90% v 60%; P < .01) and fewer hallucinations (34% v 53%; P < .01). GPT-4's overall G-PS was significantly higher (0.34 v –0.15; P < .01), indicating superior performance. CONCLUSION This study highlights the rapid improvement of generative AI in matching treatment recommendations with biomarkers in precision oncology. Although the rate of hallucinations improved in the GPT-4 model, future generative AI use in clinical care requires high levels of accuracy with minimal to no room for hallucinations. The GP-S represents a novel metric quantifying generative AI utility in health care compared with national guidelines, with potential adaptation beyond precision oncology.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Carcinoma, Non-Small-Cell LungLung NeoplasmsOncogenesArtificial IntelligenceHumansBiomarkers, TumorPrecision MedicineHigh-Throughput Nucleotide Sequencing

Substanzen

Biomarkers, Tumor

Autor:innen und Institutionen

  • University of Illinois Chicago, Chicago, IL.
  • University of Washington, Seattle, WA.
  • Kirkland & Ellis, Chicago, IL.

ORCID-Profile: Zacharie Hamilton, Aseem Aseem, Zhengjia Chen, Noor Naffakh, Natalie M Reizine, Frank Weinberg, Shikha Jain, Vijayakrishna K Gadi, Ryan H Nguyen

Zitieren

APA

Hamilton, Z., Aseem, A., Chen, Z., Naffakh, N., Reizine, N. M., Weinberg, F., Jain, S., Kessler, L. G., Gadi, V. K., Bun, C., & Nguyen, R. H. (2024). Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score. JCO Clinical Cancer Informatics. https://doi.org/10.1200/cci.24.00123

Vancouver

Hamilton Z, Aseem A, Chen Z, Naffakh N, Reizine NM, Weinberg F, et al. Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score. JCO Clinical Cancer Informatics. 2024. doi:10.1200/cci.24.00123

BibTeX
@article{Hamilton2024,
  author = {Hamilton, Zacharie and Aseem, Aseem and Chen, Zhengjia and Naffakh, Noor and Reizine, Natalie M and Weinberg, Frank and Jain, Shikha and Kessler, Larry G and Gadi, Vijayakrishna K and Bun, Christopher and Nguyen, Ryan H},
  title = {Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non–Small Cell Lung Cancer: Introducing the Generative Artificial Intelligence Performance Score},
  journal = {JCO Clinical Cancer Informatics},
  year = {2024},
  doi = {10.1200/cci.24.00123}
}
Kostenlos starten →