EUVIMEDEUVIMED entdecken
EvaluationsstudieOpen Access

Large Language Models in Summarizing Radiology Report Impressions for Lung Cancer in Chinese: Evaluation Study

Danqing Hu iD, Shanyuan Zhang iD, Qing Liu iD, Xiaofeng Zhu iD, Bing Liu iD

Journal of Medical Internet Research · 03.04.2025 · 27 · e65547

DOI: 10.2196/65547 ↗

Kurz gefasst

Evaluationsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 03.04.2025. Verfasst von Danqing Hu und 4 weiteren Autor:innen. Beteiligte Einrichtung: Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology,…. Hauptthemen laut MEDLINE: Lung Neoplasms, Natural Language Processing. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs), such as ChatGPT, have demonstrated impressive capabilities in various natural language processing tasks, particularly in text generation. However, their effectiveness in summarizing radiology report impressions remains uncertain.

Objective

This study aims to evaluate the capability of nine LLMs, that is, Tongyi Qianwen, ERNIE Bot, ChatGPT, Bard, Claude, Baichuan, ChatGLM, HuatuoGPT, and ChatGLM-Med, in summarizing Chinese radiology report impressions for lung cancer.

Methods

We collected 100 Chinese computed tomography (CT), positron emission tomography (PET)–CT, and ultrasound (US) reports each from Peking University Cancer Hospital and Institute. All these reports were from patients with suspected or confirmed lung cancer. Using these reports, we created zero-shot, one-shot, and three-shot prompts with or without complete example reports as inputs to generate impressions. We used both automatic quantitative evaluation metrics and five human evaluation metrics (completeness, correctness, conciseness, verisimilitude, and replaceability) to assess the generated impressions. Two thoracic surgeons (SZ and BL) and one radiologist (QL) compared the generated impressions with reference impressions, scoring them according to the five human evaluation metrics.

Results

In the automatic quantitative evaluation, ERNIE Bot, Tongyi Qianwen, and Claude demonstrated the best overall performance in generating impressions for CT, PET-CT, and US reports, respectively. In the human semantic evaluation, ERNIE Bot outperformed the other LLMs in terms of conciseness, verisimilitude, and replaceability on CT impression generation, while its completeness and correctness scores were comparable to those of other LLMs. Tongyi Qianwen excelled in PET-CT impression generation, with the highest scores for correctness, conciseness, verisimilitude, and replaceability. Claude achieved the best conciseness, verisimilitude, and replaceability scores on US impression generation, and its completeness and correctness scores are close to the best results obtained by other LLMs. The generated impressions were generally complete and correct but lacked conciseness and verisimilitude. Although one-shot and few-shot prompts improved conciseness and verisimilitude, clinicians noted a significant gap between the generated impressions and those written by radiologists.

Conclusions

Current LLMs can produce radiology impressions with high completeness and correctness but fall short in conciseness and verisimilitude, indicating they cannot yet fully replace impressions written by radiologists.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Lung NeoplasmsNatural Language ProcessingHumansChinaLarge Language ModelsPositron Emission Tomography Computed TomographyTomography, X-Ray Computed

Stichwörter der Autor:innen

ChatGPTevaluation studyimpression summarizationlarge language modelnatural language processingradiologistradiologyradiology reportthoracic surgeonsultrasound

Autor:innen und Institutionen

  • Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology, Nanjing, China.
  • Zhejiang Lab, Hangzhou, China.
  • Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Department of Thoracic Surgery II, Peking University Cancer Hospital and Institute, Beijing, China.
  • Department of Radiology, Peking University Cancer Hospital and Institute, Beijing, China.

ORCID-Profile: Danqing Hu, Shanyuan Zhang, Qing Liu, Xiaofeng Zhu, Bing Liu

Zitieren

APA

Hu, D., Zhang, S., Liu, Q., Zhu, X., & Liu, B. (2025). Large Language Models in Summarizing Radiology Report Impressions for Lung Cancer in Chinese: Evaluation Study. Journal of Medical Internet Research, 27, e65547. https://doi.org/10.2196/65547

Vancouver

Hu D, Zhang S, Liu Q, Zhu X, Liu B. Large Language Models in Summarizing Radiology Report Impressions for Lung Cancer in Chinese: Evaluation Study. Journal of Medical Internet Research. 2025;27:e65547. doi:10.2196/65547

BibTeX
@article{Hu2025,
  author = {Hu, Danqing and Zhang, Shanyuan and Liu, Qing and Zhu, Xiaofeng and Liu, Bing},
  title = {Large Language Models in Summarizing Radiology Report Impressions for Lung Cancer in Chinese: Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e65547},
  doi = {10.2196/65547}
}
Kostenlos starten →