Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study
JMIR Formative Research · 08.02.2024 · 8 · e32690
DOI: 10.2196/32690 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 08.02.2024. Verfasst von Jia Ji und 4 weiteren Autor:innen. Beteiligte Einrichtung: Shenzhen Institute of Information Technology, Shenzhen, China. Die Arbeit wurde bisher 48-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The automatic generation of radiology reports, which seeks to create a free-text description from a clinical radiograph, is emerging as a pivotal intersection between clinical medicine and artificial intelligence. Leveraging natural language processing technologies can accelerate report creation, enhancing health care quality and standardization. However, most existing studies have not yet fully tapped into the combined potential of advanced language and vision models.
Objective
The purpose of this study was to explore the integration of pretrained vision-language models into radiology report generation. This would enable the vision-language model to automatically convert clinical images into high-quality textual reports.
Methods
In our research, we introduced a radiology report generation model named ClinicalBLIP, building upon the foundational InstructBLIP model and refining it using clinical image-to-text data sets. A multistage fine-tuning approach via low-rank adaptation was proposed to deepen the semantic comprehension of the visual encoder and the large language model for clinical imagery. Furthermore, prior knowledge was integrated through prompt learning to enhance the precision of the reports generated. Experiments were conducted on both the IU X-RAY and MIMIC-CXR data sets, with ClinicalBLIP compared to several leading methods.
Results
Experimental results revealed that ClinicalBLIP obtained superior scores of 0.570/0.365 and 0.534/0.313 on the IU X-RAY/MIMIC-CXR test sets for the Metric for Evaluation of Translation with Explicit Ordering (METEOR) and the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) evaluations, respectively. This performance notably surpasses that of existing state-of-the-art methods. Further evaluations confirmed the effectiveness of the multistage fine-tuning and the integration of prior information, leading to substantial improvements.
Conclusions
The proposed ClinicalBLIP model demonstrated robustness and effectiveness in enhancing clinical radiology report generation, suggesting significant promise for real-world clinical applications.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Shenzhen Institute of Information Technology, Shenzhen, China.
- Peng Cheng Laboratory, Shenzhen, China.
- Harbin Institute of Technology, Shenzhen, China.
ORCID-Profile: Jia Ji, Yongshuai Hou, Xinyu Chen, Youcheng Pan, Yang Xiang
Zitieren
APA
Ji, J., Hou, Y., Chen, X., Pan, Y., & Xiang, Y. (2024). Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study. JMIR Formative Research, 8, e32690. https://doi.org/10.2196/32690
Vancouver
Ji J, Hou Y, Chen X, Pan Y, Xiang Y. Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study. JMIR Formative Research. 2024;8:e32690. doi:10.2196/32690
BibTeX
@article{Ji2024,
author = {Ji, Jia and Hou, Yongshuai and Chen, Xinyu and Pan, Youcheng and Xiang, Yang},
title = {Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study},
journal = {JMIR Formative Research},
year = {2024},
volume = {8},
pages = {e32690},
doi = {10.2196/32690}
}
EUVIMED