EUVIMEDEUVIMED entdecken
Open Access

Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study

Jia Ji iD, Yongshuai Hou iD, Xinyu Chen iD, Youcheng Pan iD, Yang Xiang iD

JMIR Formative Research · 08.02.2024 · 8 · e32690

DOI: 10.2196/32690 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 08.02.2024. Verfasst von Jia Ji und 4 weiteren Autor:innen. Beteiligte Einrichtung: Shenzhen Institute of Information Technology, Shenzhen, China. Die Arbeit wurde bisher 48-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The automatic generation of radiology reports, which seeks to create a free-text description from a clinical radiograph, is emerging as a pivotal intersection between clinical medicine and artificial intelligence. Leveraging natural language processing technologies can accelerate report creation, enhancing health care quality and standardization. However, most existing studies have not yet fully tapped into the combined potential of advanced language and vision models.

Objective

The purpose of this study was to explore the integration of pretrained vision-language models into radiology report generation. This would enable the vision-language model to automatically convert clinical images into high-quality textual reports.

Methods

In our research, we introduced a radiology report generation model named ClinicalBLIP, building upon the foundational InstructBLIP model and refining it using clinical image-to-text data sets. A multistage fine-tuning approach via low-rank adaptation was proposed to deepen the semantic comprehension of the visual encoder and the large language model for clinical imagery. Furthermore, prior knowledge was integrated through prompt learning to enhance the precision of the reports generated. Experiments were conducted on both the IU X-RAY and MIMIC-CXR data sets, with ClinicalBLIP compared to several leading methods.

Results

Experimental results revealed that ClinicalBLIP obtained superior scores of 0.570/0.365 and 0.534/0.313 on the IU X-RAY/MIMIC-CXR test sets for the Metric for Evaluation of Translation with Explicit Ordering (METEOR) and the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) evaluations, respectively. This performance notably surpasses that of existing state-of-the-art methods. Further evaluations confirmed the effectiveness of the multistage fine-tuning and the integration of prior information, leading to substantial improvements.

Conclusions

The proposed ClinicalBLIP model demonstrated robustness and effectiveness in enhancing clinical radiology report generation, suggesting significant promise for real-world clinical applications.

Schlagwörter

Stichwörter der Autor:innen

clinical imagemultistage fine-tuningprior knowledgeradiology report generationvision-language model

Autor:innen und Institutionen

  • Shenzhen Institute of Information Technology, Shenzhen, China.
  • Peng Cheng Laboratory, Shenzhen, China.
  • Harbin Institute of Technology, Shenzhen, China.

ORCID-Profile: Jia Ji, Yongshuai Hou, Xinyu Chen, Youcheng Pan, Yang Xiang

Zitieren

APA

Ji, J., Hou, Y., Chen, X., Pan, Y., & Xiang, Y. (2024). Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study. JMIR Formative Research, 8, e32690. https://doi.org/10.2196/32690

Vancouver

Ji J, Hou Y, Chen X, Pan Y, Xiang Y. Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study. JMIR Formative Research. 2024;8:e32690. doi:10.2196/32690

BibTeX
@article{Ji2024,
  author = {Ji, Jia and Hou, Yongshuai and Chen, Xinyu and Pan, Youcheng and Xiang, Yang},
  title = {Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study},
  journal = {JMIR Formative Research},
  year = {2024},
  volume = {8},
  pages = {e32690},
  doi = {10.2196/32690}
}
Kostenlos starten →