EUVIMEDEUVIMED entdecken
Crossref

MCPL: Multi-Modal Collaborative Prompt Learning for Medical Vision-Language Model

Pengyu Wang, Huaqi Zhang, Yixuan Yuan

IEEE Transactions on Medical Imaging · 2024-12 · 43 (12) · 4224-4235

DOI: 10.1109/tmi.2024.3418408 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2024-12. Verfasst von Pengyu Wang und 2 weiteren Autor:innen. Hauptthemen laut MEDLINE: Machine Learning, Image Processing, Computer-Assisted. Die Arbeit wurde bisher 30-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: Hong Kong Research Grants Council (RGC) General Research Fund.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multi-modal prompt learning is a high-performance and cost-effective learning paradigm, which learns text as well as image prompts to tune pre-trained vision-language (V-L) models like CLIP for adapting multiple downstream tasks. However, recent methods typically treat text and image prompts as independent components without considering the dependency between prompts. Moreover, extending multi-modal prompt learning into the medical field poses challenges due to a significant gap between general- and medical-domain data. To this end, we propose a Multi-modal Collaborative Prompt Learning (MCPL) pipeline to tune a frozen V-L model for aligning medical text-image representations, thereby achieving medical downstream tasks. We first construct the anatomy-pathology (AP) prompt for multi-modal prompting jointly with text and image prompts. The AP prompt introduces instance-level anatomy and pathology information, thereby making a V-L model better comprehend medical reports and images. Next, we propose graph-guided prompt collaboration module (GPCM), which explicitly establishes multi-way couplings between the AP, text, and image prompts, enabling collaborative multi-modal prompt producing and updating for more effective prompting. Finally, we develop a novel prompt configuration scheme, which attaches the AP prompt to the query and key, and the text/image prompt to the value in self-attention layers for improving the interpretability of multi-modal prompts. Extensive experiments on numerous medical classification and object detection datasets show that the proposed pipeline achieves excellent effectiveness and generalization. Compared with state-of-the-art prompt learning methods, MCPL provides a more reliable multi-modal prompt paradigm for reducing tuning costs of V-L models on medical downstream tasks. Our code: https://github.com/CUHK-AIM-Group/MCPL.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Machine LearningImage Processing, Computer-AssistedHumansAlgorithms

Förderung

  • Hong Kong Research Grants Council (RGC) General Research Fund

Zitieren

APA

Wang, P., Zhang, H., & Yuan, Y. (2024). MCPL: Multi-Modal Collaborative Prompt Learning for Medical Vision-Language Model. IEEE Transactions on Medical Imaging, 43(12), 4224-4235. https://doi.org/10.1109/tmi.2024.3418408

Vancouver

Wang P, Zhang H, Yuan Y. MCPL: Multi-Modal Collaborative Prompt Learning for Medical Vision-Language Model. IEEE Transactions on Medical Imaging. 2024;43(12):4224-4235. doi:10.1109/tmi.2024.3418408

BibTeX
@article{Wang2024,
  author = {Wang, Pengyu and Zhang, Huaqi and Yuan, Yixuan},
  title = {MCPL: Multi-Modal Collaborative Prompt Learning for Medical Vision-Language Model},
  journal = {IEEE Transactions on Medical Imaging},
  year = {2024},
  volume = {43},
  number = {12},
  pages = {4224--4235},
  doi = {10.1109/tmi.2024.3418408}
}
Kostenlos starten →