Systematic Analysis of Vision–Language Models for Medical Visual Question Answering
Multimodal Technologies and Interaction · 03.02.2026 · 10 (2) · 16
DOI: 10.3390/mti10020016 ↗Kurz gefasst
Fachartikel in der Zeitschrift Multimodal Technologies and Interaction, online erschienen am 03.02.2026. Verfasst von Muhammad Haseeb Shah und 1 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
General-purpose vision–language models (VLMs) are increasingly applied to imaging tasks, yet their reliability on medical visual question answering (Med-VQA) remains unclear. We investigate how three state-of-the-art VLMs—ViLT, BLIP, and MiniCPM-V-2—perform on radiology-focused Med-VQA when evaluated in a modality-aware manner. Using SLAKE and OmniMedVQA-Mini, we construct harmonised subsets for computed tomography (CT), magnetic resonance imaging (MRI), and X-ray, standardising schema and answer processing. We first benchmark all models in a strict zero-shot setting, then perform supervised fine-tuning on modality-specific data splits, and finally add a post-hoc semantic option-selection layer that maps free-text predictions to multiple-choice answers. Zero-shot performance is modest (exact match ≈20% for ViLT/BLIP and 0% for MiniCPM-V-2), confirming that off-the-shelf deployment is inadequate. Fine-tuning substantially improves all models, with ViLT reaching ≈80% exact match and BLIP ≈50%, while MiniCPM-V-2 lags behind. When coupled with option selection, ViLT and BLIP achieve 90–93% exact match and F1 across all modalities, corresponding to 95–97% BERTScore-F1. Our novel results show that (i) modality-specific supervision is essential for Med-VQA, and (ii) post-hoc option selection can transform strong but imperfect generative predictions into highly reliable discrete decisions on harmonised radiology benchmarks. The latter is useful for medical VLMs that combine generative responses with option or sentence selection.
Zitieren
APA
Shah, M. H., & Cuayáhuitl, H. (2026). Systematic Analysis of Vision–Language Models for Medical Visual Question Answering. Multimodal Technologies and Interaction, 10(2), 16. https://doi.org/10.3390/mti10020016
Vancouver
Shah MH, Cuayáhuitl H. Systematic Analysis of Vision–Language Models for Medical Visual Question Answering. Multimodal Technologies and Interaction. 2026;10(2):16. doi:10.3390/mti10020016
BibTeX
@article{Shah2026,
author = {Shah, Muhammad Haseeb and Cuayáhuitl, Heriberto},
title = {Systematic Analysis of Vision–Language Models for Medical Visual Question Answering},
journal = {Multimodal Technologies and Interaction},
year = {2026},
volume = {10},
number = {2},
pages = {16},
doi = {10.3390/mti10020016}
}
EUVIMED