EUVIMEDEUVIMED entdecken
Crossref

Multi-Aspect Knowledge-Enhanced Dermatological Vision–Language Pretraining With Multi-Agent Data Generation

Xieji Li, Siyuan Yan, Yingsheng Liu, H Peter Soyer, Monika Janda, Victoria Mar, Zongyuan Ge

IEEE Transactions on Medical Imaging · 2026-11 · 45 (11) · 5441-5454

DOI: 10.1109/tmi.2026.3723004 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2026-11. Verfasst von Xieji Li und 6 weiteren Autor:innen. Hauptthemen laut MEDLINE: Dermatology, Image Processing, Computer-Assisted. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 50 Literaturangaben. Gefördert von: National Health and Medical Research Council (NHMRC) Centre of Research Excellence in Skin Imaging and Precision Diagnosis, National Health and Medical Research Council (NHMRC) Synergy Grant: Roadmap Options for Melanoma Screening in Australia, NHMRC Investigator.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Vision-language pretraining (VLP) has emerged as a powerful paradigm in medical image analysis, enabling representation learning from large-scale image-text pairs without relying on expensive expertise annotations. However, these challenges are particularly pronounced when VLP is applied to dermatology. The inherent noise and sparsity of web-crawled data hinder effective pretraining, while multifaceted dermatological knowledge is challenging to exploit from long, unstructured descriptions for pretraining optimization. To address these challenges, we propose a dermatological VLP framework integrating a Multi-Agent data GENeration (MAGEN) system and Ontology-based Multi-Aspect Knowledge-Enhanced (O-MAKE) pretraining. First, MAGEN enhances data quality by synthesizing knowledge-enriched descriptions via a foundation model-assisted captioning and retrieval-based verification pipeline. Second, O-MAKE addresses the difficulty of learning from long, unstructured texts by decomposing them into distinct knowledge aspects. This facilitates fine-grained alignment at both global and patch levels, while explicitly modeling medical concept relationships through ontology-guided mechanisms. Comprehensive experiments demonstrate the effectiveness of each component, achieving state-of-the-art zero-shot performance on disease classification and cross-modal retrieval tasks across eight dermatological datasets. Our code and the augmented dataset Derm1M-AgentAug, comprising over 400K skin image-text pairs, are publicly available at https://github.com/XiejiLi/MAGEN-O-MAKE.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DermatologyImage Processing, Computer-AssistedHumansSkin DiseasesRepresentation Machine Learning

Förderung

  • National Health and Medical Research Council (NHMRC) Centre of Research Excellence in Skin Imaging and Precision Diagnosis
  • National Health and Medical Research Council (NHMRC) Synergy Grant: Roadmap Options for Melanoma Screening in Australia
  • NHMRC Investigator

Zitieren

APA

Li, X., Yan, S., Liu, Y., Soyer, H. P., Janda, M., Mar, V., & Ge, Z. (2026). Multi-Aspect Knowledge-Enhanced Dermatological Vision–Language Pretraining With Multi-Agent Data Generation. IEEE Transactions on Medical Imaging, 45(11), 5441-5454. https://doi.org/10.1109/tmi.2026.3723004

Vancouver

Li X, Yan S, Liu Y, Soyer HP, Janda M, Mar V, et al. Multi-Aspect Knowledge-Enhanced Dermatological Vision–Language Pretraining With Multi-Agent Data Generation. IEEE Transactions on Medical Imaging. 2026;45(11):5441-5454. doi:10.1109/tmi.2026.3723004

BibTeX
@article{Li2026,
  author = {Li, Xieji and Yan, Siyuan and Liu, Yingsheng and Soyer, H Peter and Janda, Monika and Mar, Victoria and Ge, Zongyuan},
  title = {Multi-Aspect Knowledge-Enhanced Dermatological Vision–Language Pretraining With Multi-Agent Data Generation},
  journal = {IEEE Transactions on Medical Imaging},
  year = {2026},
  volume = {45},
  number = {11},
  pages = {5441--5454},
  doi = {10.1109/tmi.2026.3723004}
}
Kostenlos starten →