EUVIMEDEUVIMED entdecken
Crossref

Multimodal skin disease classification using vision transformers, medical captioning, and metadata fusion: an analysis on the ISIC 2024 dataset

Aadit Shrestha iD, Aditi Palit iD

Biomedical Physics & Engineering Express · 01.04.2026 · 12 (2) · 025043

DOI: 10.1088/2057-1976/ae4eeb ↗

Kurz gefasst

Fachartikel in der Zeitschrift Biomedical Physics & Engineering Express, online erschienen am 18.03.2026. Verfasst von Aadit Shrestha und 1 weiteren Autor:innen. Beteiligte Einrichtung: Independent Researcher, Kathmandu, Nepal. Hauptthemen laut MEDLINE: Skin Diseases, Metadata, Dermoscopy, Deep Learning. Die Arbeit nennt 14 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Skin cancer and dermatological diseases are among the most prevalent global health conditions, where early and accurate diagnosis is critical for improving patient outcomes. Although deep learning models have achieved strong performance in dermoscopic image classification, many existing approaches primarily rely on visual features and make limited use of complementary clinical metadata and language-based context routinely considered by dermatologists. Recent vision–language models (VLMs), including medical-domain adaptations such as MedCLIP, have begun to show promise in dermatology; however, their integration with structured clinical metadata and the impact of different multimodal fusion strategies have not been systematically analyzed. In this work, we address the binary skin lesion classification problem by conducting a structured evaluation of MedCLIP-based multimodal embeddings combined with classical machine learning and neural classifiers. Image–text representations are extracted using MedCLIP and fused with patient metadata through early and attention-based fusion mechanisms, followed by multilayer perceptron (MLP) and ensemble classifiers. Experiments are performed on a curated subset of the ISIC 2024 dataset comprising 1,600 training and 400 test dermoscopic images with associated metadata. The proposed multimodal approach achieves an accuracy of 96% (95.7% exact) with AUROC = 0.987, outperforming unimodal baselines and demonstrating the complementary value of language and metadata for skin lesion diagnosis. This study provides a comprehensive analysis of MedCLIP-based multimodal learning in dermatology and highlights the importance of fusion design in vision–language–metadata systems for computer-aided diagnosis.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Skin DiseasesMetadataDermoscopyDeep LearningImage Processing, Computer-AssistedHumansDatabases, FactualMultilayer PerceptronsMachine LearningAlgorithmsClassification Algorithms

Stichwörter der Autor:innen

ISIC datasetMLP modelsMed-CLIPclassification problemmedical imagingskin diseasevision-language models

Autor:innen und Institutionen

  • Independent Researcher, Kathmandu, Nepal.
  • Department of Computer Science and Engineering, Indian Institute of Technology Tirupati, Tirupati, Andhra Pradesh, 517619, India.

ORCID-Profile: Aadit Shrestha, Aditi Palit

Zitieren

APA

Shrestha, A., & Palit, A. (2026). Multimodal skin disease classification using vision transformers, medical captioning, and metadata fusion: an analysis on the ISIC 2024 dataset. Biomedical Physics & Engineering Express, 12(2), 025043. https://doi.org/10.1088/2057-1976/ae4eeb

Vancouver

Shrestha A, Palit A. Multimodal skin disease classification using vision transformers, medical captioning, and metadata fusion: an analysis on the ISIC 2024 dataset. Biomedical Physics & Engineering Express. 2026;12(2):025043. doi:10.1088/2057-1976/ae4eeb

BibTeX
@article{Shrestha2026,
  author = {Shrestha, Aadit and Palit, Aditi},
  title = {Multimodal skin disease classification using vision transformers, medical captioning, and metadata fusion: an analysis on the ISIC 2024 dataset},
  journal = {Biomedical Physics & Engineering Express},
  year = {2026},
  volume = {12},
  number = {2},
  pages = {025043},
  doi = {10.1088/2057-1976/ae4eeb}
}
Kostenlos starten →