EUVIMEDEUVIMED entdecken
Crossref

Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models

Chenyu Lian, Hong-Yu Zhou, Dongyun Liang, Jing Qin, Liansheng Wang

IEEE Transactions on Medical Imaging · 2025-11 · 44 (11) · 4499-4510

DOI: 10.1109/tmi.2025.3575853 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2025-11. Verfasst von Chenyu Lian und 4 weiteren Autor:innen. Hauptthemen laut MEDLINE: Image Processing, Computer-Assisted. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Fujian Provincial Natural Science Foundation of China, Innovation and Technology Fund under Hong Kong Innovation and Technology Commission.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Medical vision-language alignment through cross-modal contrastive learning shows promising performance in image-text matching tasks, such as retrieval and zero-shot classification. However, conventional cross-modal contrastive learning (CLIP-based) methods suffer from suboptimal visual representation capabilities, which also limits their effectiveness in vision-language alignment. In contrast, although the models pretrained via multimodal masked modeling struggle with direct cross-modal matching, they excel in visual representation. To address this contradiction, we propose ALTA (ALign Through Adapting), an efficient medical vision-language alignment method that utilizes only about 8% of the trainable parameters and less than 1/5 of the computational consumption required for masked record modeling. ALTA achieves superior performance in vision-language matching tasks like retrieval and zero-shot classification by adapting the pretrained vision model from masked record modeling. Additionally, we integrate temporal-multiview radiograph inputs to enhance the information consistency between radiographs and their corresponding descriptions in reports, further improving the vision-language alignment. Experimental evaluations show that ALTA outperforms the best-performing counterpart by over 4% absolute points in text-to-image accuracy and approximately 6% absolute points in image-to-text retrieval accuracy. The adaptation of vision-language models during efficient alignment also promotes better vision and language understanding. Code is publicly available at https://github.com/DopamineLcy/ALTA.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Image Processing, Computer-AssistedHumansAlgorithmsMachine Learning

Förderung

  • National Natural Science Foundation of China
  • Fujian Provincial Natural Science Foundation of China
  • Innovation and Technology Fund under Hong Kong Innovation and Technology Commission
  • Collaborative Research with World-Leading Research Groups in The Hong Kong Polytechnic University
  • Shenzhen-Hong Kong-Macao Science and Technology Plan Project (Category C Project) under Shenzhen Municipal Science and Technology Innovation Commission

Zitieren

APA

Lian, C., Zhou, H. Y., Liang, D., Qin, J., & Wang, L. (2025). Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models. IEEE Transactions on Medical Imaging, 44(11), 4499-4510. https://doi.org/10.1109/tmi.2025.3575853

Vancouver

Lian C, Zhou HY, Liang D, Qin J, Wang L. Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models. IEEE Transactions on Medical Imaging. 2025;44(11):4499-4510. doi:10.1109/tmi.2025.3575853

BibTeX
@article{Lian2025,
  author = {Lian, Chenyu and Zhou, Hong-Yu and Liang, Dongyun and Qin, Jing and Wang, Liansheng},
  title = {Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models},
  journal = {IEEE Transactions on Medical Imaging},
  year = {2025},
  volume = {44},
  number = {11},
  pages = {4499--4510},
  doi = {10.1109/tmi.2025.3575853}
}
Kostenlos starten →