Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
IEEE Transactions on Medical Imaging · 2025-11 · 44 (11) · 4499-4510
DOI: 10.1109/tmi.2025.3575853 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2025-11. Verfasst von Chenyu Lian und 4 weiteren Autor:innen. Hauptthemen laut MEDLINE: Image Processing, Computer-Assisted. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Fujian Provincial Natural Science Foundation of China, Innovation and Technology Fund under Hong Kong Innovation and Technology Commission.
Abstract (Originalsprache)
Medical vision-language alignment through cross-modal contrastive learning shows promising performance in image-text matching tasks, such as retrieval and zero-shot classification. However, conventional cross-modal contrastive learning (CLIP-based) methods suffer from suboptimal visual representation capabilities, which also limits their effectiveness in vision-language alignment. In contrast, although the models pretrained via multimodal masked modeling struggle with direct cross-modal matching, they excel in visual representation. To address this contradiction, we propose ALTA (ALign Through Adapting), an efficient medical vision-language alignment method that utilizes only about 8% of the trainable parameters and less than 1/5 of the computational consumption required for masked record modeling. ALTA achieves superior performance in vision-language matching tasks like retrieval and zero-shot classification by adapting the pretrained vision model from masked record modeling. Additionally, we integrate temporal-multiview radiograph inputs to enhance the information consistency between radiographs and their corresponding descriptions in reports, further improving the vision-language alignment. Experimental evaluations show that ALTA outperforms the best-performing counterpart by over 4% absolute points in text-to-image accuracy and approximately 6% absolute points in image-to-text retrieval accuracy. The adaptation of vision-language models during efficient alignment also promotes better vision and language understanding. Code is publicly available at https://github.com/DopamineLcy/ALTA.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- National Natural Science Foundation of China
- Fujian Provincial Natural Science Foundation of China
- Innovation and Technology Fund under Hong Kong Innovation and Technology Commission
- Collaborative Research with World-Leading Research Groups in The Hong Kong Polytechnic University
- Shenzhen-Hong Kong-Macao Science and Technology Plan Project (Category C Project) under Shenzhen Municipal Science and Technology Innovation Commission
Zitieren
APA
Lian, C., Zhou, H. Y., Liang, D., Qin, J., & Wang, L. (2025). Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models. IEEE Transactions on Medical Imaging, 44(11), 4499-4510. https://doi.org/10.1109/tmi.2025.3575853
Vancouver
Lian C, Zhou HY, Liang D, Qin J, Wang L. Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models. IEEE Transactions on Medical Imaging. 2025;44(11):4499-4510. doi:10.1109/tmi.2025.3575853
BibTeX
@article{Lian2025,
author = {Lian, Chenyu and Zhou, Hong-Yu and Liang, Dongyun and Qin, Jing and Wang, Liansheng},
title = {Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models},
journal = {IEEE Transactions on Medical Imaging},
year = {2025},
volume = {44},
number = {11},
pages = {4499--4510},
doi = {10.1109/tmi.2025.3575853}
}
EUVIMED