Assessing the performance of ChatGPT and Bard/Gemini against radiologists for Prostate Imaging-Reporting and Data System classification based on prostate multiparametric MRI text reports
British Journal of Radiology · 2025
DOI: 10.1093/bjr/tqae236 ↗Kurz gefasst
Fachartikel in der Zeitschrift British Journal of Radiology, 2025. Verfasst von Kang-Lung Lee und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, United Kingdom. Hauptthemen laut MEDLINE: Prostatic Neoplasms, Multiparametric Magnetic Resonance Imaging, Radiology Information Systems, Magnetic Resonance Imaging. Die Arbeit wurde bisher 30-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIHR Cambridge Biomedical Research Centre.
Abstract (Originalsprache)
Objectives
Large language models (LLMs) have shown potential for clinical applications. This study assesses their ability to assign Prostate Imaging-Reporting and Data System (PI-RADS) categories based on clinical text reports.
Methods
One hundred consecutive biopsy-naïve patients’ multiparametric prostate MRI reports were independently classified by 2 uroradiologists, ChatGPT-3.5 (GPT-3.5), ChatGPT-4o mini (GPT-4), Bard, and Gemini. Original report classifications were considered definitive.
Results
Out of 100 MRIs, 52 were originally reported as PI-RADS 1-2, 9 PI-RADS 3, 19 PI-RADS 4, and 20 PI-RADS 5. Radiologists demonstrated 95% and 90% accuracy, while GPT-3.5 and Bard both achieved 67%. Accuracy of the updated versions of LLMs increased to 83% (GTP-4) and 79% (Gemini), respectively. In low suspicion studies (PI-RADS 1-2), Bard and Gemini (F1: 0.94, 0.98, respectively) outperformed GPT-3.5 and GTP-4 (F1:0.77, 0.94, respectively), whereas for high probability MRIs (PI-RADS 4-5), GPT-3.5 and GTP-4 (F1: 0.95, 0.98, respectively) outperformed Bard and Gemini (F1: 0.71, 0.87, respectively). Bard assigned a non-existent PI-RADS 6 “hallucination” for 2 patients. Inter-reader agreements (Κ) between the original reports and the senior radiologist, junior radiologist, GPT-3.5, GTP-4, BARD, and Gemini were 0.93, 0.84, 0.65, 0.86, 0.57, and 0.81, respectively.
Conclusions
Radiologists demonstrated high accuracy in PI-RADS classification based on text reports, while GPT-3.5 and Bard exhibited poor performance. GTP-4 and Gemini demonstrated improved performance compared to their predecessors. Advances in knowledge This study highlights the limitations of LLMs in accurately classifying PI-RADS categories from clinical text reports. While the performance of LLMs has improved with newer versions, caution is warranted before integrating such technologies into clinical practice.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, United Kingdom.
- Department of Radiology, Cambridge University Hospitals NHS Foundation Trust Addenbrooke's Hospital, Cambridge CB2 0QQ, United Kingdom.
- Department of Radiology, Taipei Veterans General Hospital, Taipei 112, Taiwan.
- School of Medicine, National Yang Ming Chiao Tung University, Taipei 112, Taiwan.
ORCID-Profile: Kang-Lung Lee, Tristan Barrett
Förderung
- NIHR Cambridge Biomedical Research Centre
Zitieren
APA
Lee, K. L., Kessler, D. A., Caglic, I., Kuo, Y. H., Shaida, N., & Barrett, T. (2025). Assessing the performance of ChatGPT and Bard/Gemini against radiologists for Prostate Imaging-Reporting and Data System classification based on prostate multiparametric MRI text reports. British Journal of Radiology. https://doi.org/10.1093/bjr/tqae236
Vancouver
Lee KL, Kessler DA, Caglic I, Kuo YH, Shaida N, Barrett T. Assessing the performance of ChatGPT and Bard/Gemini against radiologists for Prostate Imaging-Reporting and Data System classification based on prostate multiparametric MRI text reports. British Journal of Radiology. 2025. doi:10.1093/bjr/tqae236
BibTeX
@article{Lee2025,
author = {Lee, Kang-Lung and Kessler, Dimitri A and Caglic, Iztok and Kuo, Yi-Hsin and Shaida, Nadeem and Barrett, Tristan},
title = {Assessing the performance of ChatGPT and Bard/Gemini against radiologists for Prostate Imaging-Reporting and Data System classification based on prostate multiparametric MRI text reports},
journal = {British Journal of Radiology},
year = {2025},
doi = {10.1093/bjr/tqae236}
}
EUVIMED