Large Language Models in Summarizing Radiology Report Impressions for Lung Cancer in Chinese: Evaluation Study
Journal of Medical Internet Research · 2025-04-03 · 27 · e65547
DOI: 10.2196/65547 ↗At a glance
Evaluationsstudie in Journal of Medical Internet Research, published online on 2025-04-03. Written by Danqing Hu and 4 co-authors. Institution: Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology,…. Major MEDLINE topics: Lung Neoplasms, Natural Language Processing. Cited 35 times so far (Crossref); 43 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Large language models (LLMs), such as ChatGPT, have demonstrated impressive capabilities in various natural language processing tasks, particularly in text generation. However, their effectiveness in summarizing radiology report impressions remains uncertain.
Objective
This study aims to evaluate the capability of nine LLMs, that is, Tongyi Qianwen, ERNIE Bot, ChatGPT, Bard, Claude, Baichuan, ChatGLM, HuatuoGPT, and ChatGLM-Med, in summarizing Chinese radiology report impressions for lung cancer.
Methods
We collected 100 Chinese computed tomography (CT), positron emission tomography (PET)–CT, and ultrasound (US) reports each from Peking University Cancer Hospital and Institute. All these reports were from patients with suspected or confirmed lung cancer. Using these reports, we created zero-shot, one-shot, and three-shot prompts with or without complete example reports as inputs to generate impressions. We used both automatic quantitative evaluation metrics and five human evaluation metrics (completeness, correctness, conciseness, verisimilitude, and replaceability) to assess the generated impressions. Two thoracic surgeons (SZ and BL) and one radiologist (QL) compared the generated impressions with reference impressions, scoring them according to the five human evaluation metrics.
Results
In the automatic quantitative evaluation, ERNIE Bot, Tongyi Qianwen, and Claude demonstrated the best overall performance in generating impressions for CT, PET-CT, and US reports, respectively. In the human semantic evaluation, ERNIE Bot outperformed the other LLMs in terms of conciseness, verisimilitude, and replaceability on CT impression generation, while its completeness and correctness scores were comparable to those of other LLMs. Tongyi Qianwen excelled in PET-CT impression generation, with the highest scores for correctness, conciseness, verisimilitude, and replaceability. Claude achieved the best conciseness, verisimilitude, and replaceability scores on US impression generation, and its completeness and correctness scores are close to the best results obtained by other LLMs. The generated impressions were generally complete and correct but lacked conciseness and verisimilitude. Although one-shot and few-shot prompts improved conciseness and verisimilitude, clinicians noted a significant gap between the generated impressions and those written by radiologists.
Conclusions
Current LLMs can produce radiology impressions with high completeness and correctness but fall short in conciseness and verisimilitude, indicating they cannot yet fully replace impressions written by radiologists.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology, Nanjing, China.
- Zhejiang Lab, Hangzhou, China.
- Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Department of Thoracic Surgery II, Peking University Cancer Hospital and Institute, Beijing, China.
- Department of Radiology, Peking University Cancer Hospital and Institute, Beijing, China.
ORCID profiles: Danqing Hu, Shanyuan Zhang, Qing Liu, Xiaofeng Zhu, Bing Liu
Cite
APA
Hu, D., Zhang, S., Liu, Q., Zhu, X., & Liu, B. (2025). Large Language Models in Summarizing Radiology Report Impressions for Lung Cancer in Chinese: Evaluation Study. Journal of Medical Internet Research, 27, e65547. https://doi.org/10.2196/65547
Vancouver
Hu D, Zhang S, Liu Q, Zhu X, Liu B. Large Language Models in Summarizing Radiology Report Impressions for Lung Cancer in Chinese: Evaluation Study. Journal of Medical Internet Research. 2025;27:e65547. doi:10.2196/65547
BibTeX
@article{Hu2025,
author = {Hu, Danqing and Zhang, Shanyuan and Liu, Qing and Zhu, Xiaofeng and Liu, Bing},
title = {Large Language Models in Summarizing Radiology Report Impressions for Lung Cancer in Chinese: Evaluation Study},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e65547},
doi = {10.2196/65547}
}
EUVIMED