EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Assessing the performance of ChatGPT and Bard/Gemini against radiologists for Prostate Imaging-Reporting and Data System classification based on prostate multiparametric MRI text reports

Kang-Lung Lee iD, Dimitri A Kessler, Iztok Caglic, Yi-Hsin Kuo, Nadeem Shaida, Tristan Barrett iD

British Journal of Radiology · 2025

DOI: 10.1093/bjr/tqae236 ↗

Kurz gefasst

Fachartikel in der Zeitschrift British Journal of Radiology, 2025. Verfasst von Kang-Lung Lee und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, United Kingdom. Hauptthemen laut MEDLINE: Prostatic Neoplasms, Multiparametric Magnetic Resonance Imaging, Radiology Information Systems, Magnetic Resonance Imaging. Die Arbeit wurde bisher 30-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIHR Cambridge Biomedical Research Centre.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Large language models (LLMs) have shown potential for clinical applications. This study assesses their ability to assign Prostate Imaging-Reporting and Data System (PI-RADS) categories based on clinical text reports.

Methods

One hundred consecutive biopsy-naïve patients’ multiparametric prostate MRI reports were independently classified by 2 uroradiologists, ChatGPT-3.5 (GPT-3.5), ChatGPT-4o mini (GPT-4), Bard, and Gemini. Original report classifications were considered definitive.

Results

Out of 100 MRIs, 52 were originally reported as PI-RADS 1-2, 9 PI-RADS 3, 19 PI-RADS 4, and 20 PI-RADS 5. Radiologists demonstrated 95% and 90% accuracy, while GPT-3.5 and Bard both achieved 67%. Accuracy of the updated versions of LLMs increased to 83% (GTP-4) and 79% (Gemini), respectively. In low suspicion studies (PI-RADS 1-2), Bard and Gemini (F1: 0.94, 0.98, respectively) outperformed GPT-3.5 and GTP-4 (F1:0.77, 0.94, respectively), whereas for high probability MRIs (PI-RADS 4-5), GPT-3.5 and GTP-4 (F1: 0.95, 0.98, respectively) outperformed Bard and Gemini (F1: 0.71, 0.87, respectively). Bard assigned a non-existent PI-RADS 6 “hallucination” for 2 patients. Inter-reader agreements (Κ) between the original reports and the senior radiologist, junior radiologist, GPT-3.5, GTP-4, BARD, and Gemini were 0.93, 0.84, 0.65, 0.86, 0.57, and 0.81, respectively.

Conclusions

Radiologists demonstrated high accuracy in PI-RADS classification based on text reports, while GPT-3.5 and Bard exhibited poor performance. GTP-4 and Gemini demonstrated improved performance compared to their predecessors. Advances in knowledge This study highlights the limitations of LLMs in accurately classifying PI-RADS categories from clinical text reports. While the performance of LLMs has improved with newer versions, caution is warranted before integrating such technologies into clinical practice.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Prostatic NeoplasmsMultiparametric Magnetic Resonance ImagingRadiology Information SystemsMagnetic Resonance ImagingHumansMaleAgedMiddle AgedRadiologistsProstateGenerative Artificial Intelligence

Stichwörter der Autor:innen

BardChatGPTGeminiPI-RADSlarge language modelprostate MRI

Autor:innen und Institutionen

  • Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, United Kingdom.
  • Department of Radiology, Cambridge University Hospitals NHS Foundation Trust Addenbrooke's Hospital, Cambridge CB2 0QQ, United Kingdom.
  • Department of Radiology, Taipei Veterans General Hospital, Taipei 112, Taiwan.
  • School of Medicine, National Yang Ming Chiao Tung University, Taipei 112, Taiwan.

ORCID-Profile: Kang-Lung Lee, Tristan Barrett

Förderung

  • NIHR Cambridge Biomedical Research Centre

Zitieren

APA

Lee, K. L., Kessler, D. A., Caglic, I., Kuo, Y. H., Shaida, N., & Barrett, T. (2025). Assessing the performance of ChatGPT and Bard/Gemini against radiologists for Prostate Imaging-Reporting and Data System classification based on prostate multiparametric MRI text reports. British Journal of Radiology. https://doi.org/10.1093/bjr/tqae236

Vancouver

Lee KL, Kessler DA, Caglic I, Kuo YH, Shaida N, Barrett T. Assessing the performance of ChatGPT and Bard/Gemini against radiologists for Prostate Imaging-Reporting and Data System classification based on prostate multiparametric MRI text reports. British Journal of Radiology. 2025. doi:10.1093/bjr/tqae236

BibTeX
@article{Lee2025,
  author = {Lee, Kang-Lung and Kessler, Dimitri A and Caglic, Iztok and Kuo, Yi-Hsin and Shaida, Nadeem and Barrett, Tristan},
  title = {Assessing the performance of ChatGPT and Bard/Gemini against radiologists for Prostate Imaging-Reporting and Data System classification based on prostate multiparametric MRI text reports},
  journal = {British Journal of Radiology},
  year = {2025},
  doi = {10.1093/bjr/tqae236}
}
Kostenlos starten →