Multimodal skin disease classification using vision transformers, medical captioning, and metadata fusion: an analysis on the ISIC 2024 dataset
Biomedical Physics & Engineering Express · 01.04.2026 · 12 (2) · 025043
DOI: 10.1088/2057-1976/ae4eeb ↗Kurz gefasst
Fachartikel in der Zeitschrift Biomedical Physics & Engineering Express, online erschienen am 18.03.2026. Verfasst von Aadit Shrestha und 1 weiteren Autor:innen. Beteiligte Einrichtung: Independent Researcher, Kathmandu, Nepal. Hauptthemen laut MEDLINE: Skin Diseases, Metadata, Dermoscopy, Deep Learning. Die Arbeit nennt 14 Literaturangaben.
Abstract (Originalsprache)
Skin cancer and dermatological diseases are among the most prevalent global health conditions, where early and accurate diagnosis is critical for improving patient outcomes. Although deep learning models have achieved strong performance in dermoscopic image classification, many existing approaches primarily rely on visual features and make limited use of complementary clinical metadata and language-based context routinely considered by dermatologists. Recent vision–language models (VLMs), including medical-domain adaptations such as MedCLIP, have begun to show promise in dermatology; however, their integration with structured clinical metadata and the impact of different multimodal fusion strategies have not been systematically analyzed. In this work, we address the binary skin lesion classification problem by conducting a structured evaluation of MedCLIP-based multimodal embeddings combined with classical machine learning and neural classifiers. Image–text representations are extracted using MedCLIP and fused with patient metadata through early and attention-based fusion mechanisms, followed by multilayer perceptron (MLP) and ensemble classifiers. Experiments are performed on a curated subset of the ISIC 2024 dataset comprising 1,600 training and 400 test dermoscopic images with associated metadata. The proposed multimodal approach achieves an accuracy of 96% (95.7% exact) with AUROC = 0.987, outperforming unimodal baselines and demonstrating the complementary value of language and metadata for skin lesion diagnosis. This study provides a comprehensive analysis of MedCLIP-based multimodal learning in dermatology and highlights the importance of fusion design in vision–language–metadata systems for computer-aided diagnosis.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Independent Researcher, Kathmandu, Nepal.
- Department of Computer Science and Engineering, Indian Institute of Technology Tirupati, Tirupati, Andhra Pradesh, 517619, India.
ORCID-Profile: Aadit Shrestha, Aditi Palit
Zitieren
APA
Shrestha, A., & Palit, A. (2026). Multimodal skin disease classification using vision transformers, medical captioning, and metadata fusion: an analysis on the ISIC 2024 dataset. Biomedical Physics & Engineering Express, 12(2), 025043. https://doi.org/10.1088/2057-1976/ae4eeb
Vancouver
Shrestha A, Palit A. Multimodal skin disease classification using vision transformers, medical captioning, and metadata fusion: an analysis on the ISIC 2024 dataset. Biomedical Physics & Engineering Express. 2026;12(2):025043. doi:10.1088/2057-1976/ae4eeb
BibTeX
@article{Shrestha2026,
author = {Shrestha, Aadit and Palit, Aditi},
title = {Multimodal skin disease classification using vision transformers, medical captioning, and metadata fusion: an analysis on the ISIC 2024 dataset},
journal = {Biomedical Physics & Engineering Express},
year = {2026},
volume = {12},
number = {2},
pages = {025043},
doi = {10.1088/2057-1976/ae4eeb}
}
EUVIMED