EUVIMEDObjevte EUVIMED
Open access

Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study

Jia Ji iD, Yongshuai Hou iD, Xinyu Chen iD, Youcheng Pan iD, Yang Xiang iD

JMIR Formative Research · 2024-02-08 · 8 · e32690

DOI: 10.2196/32690 ↗

At a glance

Journal article in JMIR Formative Research, published online on 2024-02-08. Written by Jia Ji and 4 co-authors. Institution: Shenzhen Institute of Information Technology, Shenzhen, China. Cited 48 times so far (Crossref); 38 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

The automatic generation of radiology reports, which seeks to create a free-text description from a clinical radiograph, is emerging as a pivotal intersection between clinical medicine and artificial intelligence. Leveraging natural language processing technologies can accelerate report creation, enhancing health care quality and standardization. However, most existing studies have not yet fully tapped into the combined potential of advanced language and vision models.

Objective

The purpose of this study was to explore the integration of pretrained vision-language models into radiology report generation. This would enable the vision-language model to automatically convert clinical images into high-quality textual reports.

Methods

In our research, we introduced a radiology report generation model named ClinicalBLIP, building upon the foundational InstructBLIP model and refining it using clinical image-to-text data sets. A multistage fine-tuning approach via low-rank adaptation was proposed to deepen the semantic comprehension of the visual encoder and the large language model for clinical imagery. Furthermore, prior knowledge was integrated through prompt learning to enhance the precision of the reports generated. Experiments were conducted on both the IU X-RAY and MIMIC-CXR data sets, with ClinicalBLIP compared to several leading methods.

Results

Experimental results revealed that ClinicalBLIP obtained superior scores of 0.570/0.365 and 0.534/0.313 on the IU X-RAY/MIMIC-CXR test sets for the Metric for Evaluation of Translation with Explicit Ordering (METEOR) and the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) evaluations, respectively. This performance notably surpasses that of existing state-of-the-art methods. Further evaluations confirmed the effectiveness of the multistage fine-tuning and the integration of prior information, leading to substantial improvements.

Conclusions

The proposed ClinicalBLIP model demonstrated robustness and effectiveness in enhancing clinical radiology report generation, suggesting significant promise for real-world clinical applications.

Subjects

Author keywords

clinical imagemultistage fine-tuningprior knowledgeradiology report generationvision-language model

Authors and institutions

  • Shenzhen Institute of Information Technology, Shenzhen, China.
  • Peng Cheng Laboratory, Shenzhen, China.
  • Harbin Institute of Technology, Shenzhen, China.

ORCID profiles: Jia Ji, Yongshuai Hou, Xinyu Chen, Youcheng Pan, Yang Xiang

Cite

APA

Ji, J., Hou, Y., Chen, X., Pan, Y., & Xiang, Y. (2024). Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study. JMIR Formative Research, 8, e32690. https://doi.org/10.2196/32690

Vancouver

Ji J, Hou Y, Chen X, Pan Y, Xiang Y. Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study. JMIR Formative Research. 2024;8:e32690. doi:10.2196/32690

BibTeX
@article{Ji2024,
  author = {Ji, Jia and Hou, Yongshuai and Chen, Xinyu and Pan, Youcheng and Xiang, Yang},
  title = {Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study},
  journal = {JMIR Formative Research},
  year = {2024},
  volume = {8},
  pages = {e32690},
  doi = {10.2196/32690}
}
Start for free →