EUVIMEDEUVIMED entdecken
Crossref

Large Language Models Encode Radiation Oncology Domain Knowledge: Performance on the American College of Radiology Standardized Examination

Nikhil G. Thaker, Navid Redjal, Arturo Loaiza-Bonilla, David Penberthy, Tim Showalter, Ajay Choudhri, Shirnett Williamson, Gautam Thaker, Chirag Shah, Matthew C. Ward, Mihir Thaker, Michael Arcaro

AI in Precision Oncology · 2024

DOI: 10.1089/aipo.2023.0007 ↗

Kurz gefasst

Fachartikel in der Zeitschrift AI in Precision Oncology, 2024. Verfasst von Nikhil G. Thaker und 11 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 26 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

The integration of large language models (LLMs) into medical education will represent a significant paradigm shift, offering transformative potential in how medical knowledge is accessed and assimilated. These models have not yet been systematically trained or validated on complex subspecialty medical examinations. This study explores the performance of seven major LLMs in radiation oncology. Materials and Methods: The 2021 American College of Radiology (ACR) Radiation Oncology In-Training Examination (TXIT) was used to evaluate the performance of various LLMs: OpenAI's GPT-3.5-turbo, GPT-4, GPT-4-turbo, Meta's Llama-2 models (7 billion, 13 billion, and 70 billion parameter models), and Google's PaLM-2-text-bison. The ACR provided publicly available national scoring for this examination. The examination comprised 300 questions across four major domains, including clinical, biology, physics, and statistics. The examination was processed through each LLM through application programming interface. LLM-generated answers were analyzed by domains and compared with radiation oncology trainee performance. The total cost of token inputs and outputs were aggregated and analyzed.

Results

LLMs showed varied performance, with OpenAI's GPT-4-turbo leading with 74.2% correct answers and all three Llama-2 models underperforming (ranging between 26.2% and 43.3% correct). LLMs generally excelled in the statistics domain (93.0–100%) but were less effective in clinical areas (37.0–68.0%), with the exception of GPT-4-turbo that performed comparably (68.0%) with upper-level radiation oncology trainees (PGY4–5 64.1–68.3%) and superiorly with lower-level trainees (PGY2–3 51.6–61.6%). Notably, GPT-4-turbo demonstrated 7.0% clinical improvement over its predecessor GPT-4. LLMs scored the lowest in gastrointestinal, genitourinary, and gynecology and highest in bone and soft tissue, central nervous system, and head and neck. Overall costs of LLM inputs and outputs were modest at $2.63 across all seven models.

Conclusion

GPT-4-turbo demonstrates clinical accuracy comparable with upper-level and superior with lower-level trainees. Score discrepancies across disease site domains may be due to data availability, complexity of medical conditions, and quality and quantity of training data sets. Future research will need to evaluate the performance of models that are fine-tune trained in clinical oncology. This study also underscores the need for rigorous validation of LLM-generated information against established medical literature and expert consensus, necessitating expert oversight in their application in medical education and practice.

Zitieren

APA

Thaker, N. G., Redjal, N., Loaiza-Bonilla, A., Penberthy, D., Showalter, T., Choudhri, A., Williamson, S., Thaker, G., Shah, C., Ward, M. C., Thaker, M., & Arcaro, M. (2024). Large Language Models Encode Radiation Oncology Domain Knowledge: Performance on the American College of Radiology Standardized Examination. AI in Precision Oncology. https://doi.org/10.1089/aipo.2023.0007

Vancouver

Thaker NG, Redjal N, Loaiza-Bonilla A, Penberthy D, Showalter T, Choudhri A, et al. Large Language Models Encode Radiation Oncology Domain Knowledge: Performance on the American College of Radiology Standardized Examination. AI in Precision Oncology. 2024. doi:10.1089/aipo.2023.0007

BibTeX
@article{Thaker2024,
  author = {Thaker, Nikhil G. and Redjal, Navid and Loaiza-Bonilla, Arturo and Penberthy, David and Showalter, Tim and Choudhri, Ajay and Williamson, Shirnett and Thaker, Gautam and Shah, Chirag and Ward, Matthew C. and Thaker, Mihir and Arcaro, Michael},
  title = {Large Language Models Encode Radiation Oncology Domain Knowledge: Performance on the American College of Radiology Standardized Examination},
  journal = {AI in Precision Oncology},
  year = {2024},
  doi = {10.1089/aipo.2023.0007}
}
Kostenlos starten →