EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options

Peilong Wang, Jason Holmes, Zhengliang Liu, Dequan Chen, Tianming Liu, Jiajian Shen, Wei Liu

Frontiers in Oncology · 23.05.2025 · 15

DOI: 10.3389/fonc.2025.1557064 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Oncology, online erschienen am 23.05.2025. Verfasst von Peilong Wang und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ, United States. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NCI NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

We present an updated study evaluating the performance of large language models (LLMs) in answering radiation oncology physics questions, focusing on the recently released models.

Methods

A set of 100 multiple-choice radiation oncology physics questions, previously created by a well-experienced physicist, was used for this study. The answer options of the questions were randomly shuffled to create “new” exam sets. Five LLMs – OpenAI o1-preview, GPT-4o, LLaMA 3.1 (405B), Gemini 1.5 Pro, and Claude 3.5 Sonnet – with the versions released before September 30, 2024, were queried using these new exam sets. To evaluate their deductive reasoning ability, the correct answer options in the questions were replaced with “None of the above.” Then, the explain-first and step-by-step instruction prompts were used to test if this strategy improved their reasoning ability. The performance of the LLMs was compared with the answers from medical physicists.

Results

All models demonstrated expert-level performance on these questions, with o1-preview even surpassing medical physicists with a majority vote. When replacing the correct answer options with ‘None of the above’, all models exhibited a considerable decline in performance, suggesting room for improvement. The explain-first and step-by-step instruction prompts helped enhance the reasoning ability of the LLaMA 3.1 (405B), Gemini 1.5 Pro, and Claude 3.5 Sonnet models.

Conclusion

These recently released LLMs demonstrated expert-level performance in answering radiation oncology physics questions.

Schlagwörter

Stichwörter der Autor:innen

augmentationevaluationlarge language model (LLM)physicsradiation oncology

Autor:innen und Institutionen

  • Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ, United States.
  • School of Computing, University of Georgia, Athens, GA, United States.
  • Department of Radiology, Mayo Clinic, Rochester, MN, United States.

Förderung

  • NCI NIH HHS

Zitieren

APA

Wang, P., Holmes, J., Liu, Z., Chen, D., Liu, T., Shen, J., & Liu, W. (2025). A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options. Frontiers in Oncology, 15. https://doi.org/10.3389/fonc.2025.1557064

Vancouver

Wang P, Holmes J, Liu Z, Chen D, Liu T, Shen J, et al. A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options. Frontiers in Oncology. 2025;15. doi:10.3389/fonc.2025.1557064

BibTeX
@article{Wang2025,
  author = {Wang, Peilong and Holmes, Jason and Liu, Zhengliang and Chen, Dequan and Liu, Tianming and Shen, Jiajian and Liu, Wei},
  title = {A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options},
  journal = {Frontiers in Oncology},
  year = {2025},
  volume = {15},
  doi = {10.3389/fonc.2025.1557064}
}
Kostenlos starten →