EUVIMEDScopri EUVIMED
QuerschnittstudieOpen access

Evaluating Large Language Models for Automated Reporting and Data Systems Categorization: Cross-Sectional Study

Qingxia Wu iD, Qingxia Wu iD, Huali Li iD, Yan Wang iD, Yan Bai iD, Yaping Wu iD, Xuan Yu iD, Xiaodong Li iD, Pei Dong iD, Jon Xue iD, Dinggang Shen iD, Meiyun Wang iD

JMIR Medical Informatics · 2024-07-17 · 12 · e55799

DOI: 10.2196/55799 ↗

At a glance

Querschnittstudie in JMIR Medical Informatics, published online on 2024-07-17. Written by Qingxia Wu and 11 co-authors. Institution: Department of Medical Imaging, Henan Provincial People's Hospital & People's Hospital of Zhengzhou University, Zhengzhou, China. Cited 22 times so far (Crossref); 29 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Large language models show promise for improving radiology workflows, but their performance on structured radiological tasks such as Reporting and Data Systems (RADS) categorization remains unexplored.

Objective

This study aims to evaluate 3 large language model chatbots—Claude-2, GPT-3.5, and GPT-4—on assigning RADS categories to radiology reports and assess the impact of different prompting strategies.

Methods

This cross-sectional study compared 3 chatbots using 30 radiology reports (10 per RADS criteria), using a 3-level prompting strategy: zero-shot, few-shot, and guideline PDF-informed prompts. The cases were grounded in Liver Imaging Reporting & Data System (LI-RADS) version 2018, Lung CT (computed tomography) Screening Reporting & Data System (Lung-RADS) version 2022, and Ovarian-Adnexal Reporting & Data System (O-RADS) magnetic resonance imaging, meticulously prepared by board-certified radiologists. Each report underwent 6 assessments. Two blinded reviewers assessed the chatbots’ response at patient-level RADS categorization and overall ratings. The agreement across repetitions was assessed using Fleiss κ.

Results

Claude-2 achieved the highest accuracy in overall ratings with few-shot prompts and guideline PDFs (prompt-2), attaining 57% (17/30) average accuracy over 6 runs and 50% (15/30) accuracy with k-pass voting. Without prompt engineering, all chatbots performed poorly. The introduction of a structured exemplar prompt (prompt-1) increased the accuracy of overall ratings for all chatbots. Providing prompt-2 further improved Claude-2’s performance, an enhancement not replicated by GPT-4. The interrun agreement was substantial for Claude-2 (k=0.66 for overall rating and k=0.69 for RADS categorization), fair for GPT-4 (k=0.39 for both), and fair for GPT-3.5 (k=0.21 for overall rating and k=0.39 for RADS categorization). All chatbots showed significantly higher accuracy with LI-RADS version 2018 than with Lung-RADS version 2022 and O-RADS (P

Subjects

Author keywords

ChatGPTLI-RADSLung-RADSO-RADSRadiology Reporting and Data Systemsaccuracycategorizationchatbotchatbotslarge language modelrecommendationrecommendations

Authors and institutions

  • Department of Medical Imaging, Henan Provincial People's Hospital & People's Hospital of Zhengzhou University, Zhengzhou, China.
  • Research Intelligence Department, Beijing United Imaging Research Institute of Intelligent Imaging, Beijing, China.
  • Research and Collaboration, United Imaging Intelligence (Beijing) Co, Ltd, Beijing, China.
  • Department of Radiology, Luoyang Central Hospital, Luoyang, China.
  • Research and Collaboration, Shanghai United Imaging Intelligence Co, Ltd, Shanghai, China.
  • School of Biomedical Engineering, Shanghai Tech University, Shanghai, China.

ORCID profiles: Qingxia Wu, Qingxia Wu, Huali Li, Yan Wang, Yan Bai, Yaping Wu, Xuan Yu, Xiaodong Li, Pei Dong, Jon Xue, Dinggang Shen, Meiyun Wang

Cite

APA

Wu, Q., Wu, Q., Li, H., Wang, Y., Bai, Y., Wu, Y., Yu, X., Li, X., Dong, P., Xue, J., Shen, D., & Wang, M. (2024). Evaluating Large Language Models for Automated Reporting and Data Systems Categorization: Cross-Sectional Study. JMIR Medical Informatics, 12, e55799. https://doi.org/10.2196/55799

Vancouver

Wu Q, Wu Q, Li H, Wang Y, Bai Y, Wu Y, et al. Evaluating Large Language Models for Automated Reporting and Data Systems Categorization: Cross-Sectional Study. JMIR Medical Informatics. 2024;12:e55799. doi:10.2196/55799

BibTeX
@article{Wu2024,
  author = {Wu, Qingxia and Wu, Qingxia and Li, Huali and Wang, Yan and Bai, Yan and Wu, Yaping and Yu, Xuan and Li, Xiaodong and Dong, Pei and Xue, Jon and Shen, Dinggang and Wang, Meiyun},
  title = {Evaluating Large Language Models for Automated Reporting and Data Systems Categorization: Cross-Sectional Study},
  journal = {JMIR Medical Informatics},
  year = {2024},
  volume = {12},
  pages = {e55799},
  doi = {10.2196/55799}
}
Start for free →