EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Beyond images: an integrative multi-modal approach to chest x-ray report generation

Nurbanu Aksoy, Serge Sharoff, Selcuk Baser, Nishant Ravikumar, Alejandro F Frangi

Frontiers in Radiology · 15.02.2024 · 4

DOI: 10.3389/fradi.2024.1339612 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Radiology, online erschienen am 15.02.2024. Verfasst von Nurbanu Aksoy und 4 weiteren Autor:innen. Beteiligte Einrichtung: Center for Computational Imaging & Simulation Technologies in Biomedicine, School of Computing, University of Leeds, Leeds, United Kingdom. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Milli Eğitim Bakanliği, University of Leeds.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Image-to-text radiology report generation aims to automatically produce radiology reports that describe the findings in medical images. Most existing methods focus solely on the image data, disregarding the other patient information accessible to radiologists. In this paper, we present a novel multi-modal deep neural network framework for generating chest x-rays reports by integrating structured patient data, such as vital signs and symptoms, alongside unstructured clinical notes. We introduce a conditioned cross-multi-head attention module to fuse these heterogeneous data modalities, bridging the semantic gap between visual and textual data. Experiments demonstrate substantial improvements from using additional modalities compared to relying on images alone. Notably, our model achieves the highest reported performance on the ROUGE-L metric compared to relevant state-of-the-art models in the literature. Furthermore, we employed both human evaluation and clinical semantic similarity measurement alongside word-overlap metrics to improve the depth of quantitative analysis. A human evaluation, conducted by a board-certified radiologist, confirms the model’s accuracy in identifying high-level findings, however, it also highlights that more improvement is needed to capture nuanced details and clinical context.

Schlagwörter

Stichwörter der Autor:innen

cross attentiondeep learningmulti-modal datareport generationtransformersx-ray

Autor:innen und Institutionen

  • Center for Computational Imaging & Simulation Technologies in Biomedicine, School of Computing, University of Leeds, Leeds, United Kingdom.
  • School of Languages, University of Leeds, Leeds, United Kingdom.
  • Kastamonu Training and Research Hospital, Kastamonu, Türkiye.
  • Medical Imaging Research Centre, KU Leuven, Leuven, Belgium.
  • Alan Turing Institute, London, United Kingdom.

Förderung

  • Milli Eğitim Bakanliği
  • University of Leeds

Zitieren

APA

Aksoy, N., Sharoff, S., Baser, S., Ravikumar, N., & Frangi, A. F. (2024). Beyond images: an integrative multi-modal approach to chest x-ray report generation. Frontiers in Radiology, 4. https://doi.org/10.3389/fradi.2024.1339612

Vancouver

Aksoy N, Sharoff S, Baser S, Ravikumar N, Frangi AF. Beyond images: an integrative multi-modal approach to chest x-ray report generation. Frontiers in Radiology. 2024;4. doi:10.3389/fradi.2024.1339612

BibTeX
@article{Aksoy2024,
  author = {Aksoy, Nurbanu and Sharoff, Serge and Baser, Selcuk and Ravikumar, Nishant and Frangi, Alejandro F},
  title = {Beyond images: an integrative multi-modal approach to chest x-ray report generation},
  journal = {Frontiers in Radiology},
  year = {2024},
  volume = {4},
  doi = {10.3389/fradi.2024.1339612}
}
Kostenlos starten →