Multi-Aspect Knowledge-Enhanced Dermatological Vision–Language Pretraining With Multi-Agent Data Generation
IEEE Transactions on Medical Imaging · 2026-11 · 45 (11) · 5441-5454
DOI: 10.1109/tmi.2026.3723004 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2026-11. Verfasst von Xieji Li und 6 weiteren Autor:innen. Hauptthemen laut MEDLINE: Dermatology, Image Processing, Computer-Assisted. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 50 Literaturangaben. Gefördert von: National Health and Medical Research Council (NHMRC) Centre of Research Excellence in Skin Imaging and Precision Diagnosis, National Health and Medical Research Council (NHMRC) Synergy Grant: Roadmap Options for Melanoma Screening in Australia, NHMRC Investigator.
Abstract (Originalsprache)
Vision-language pretraining (VLP) has emerged as a powerful paradigm in medical image analysis, enabling representation learning from large-scale image-text pairs without relying on expensive expertise annotations. However, these challenges are particularly pronounced when VLP is applied to dermatology. The inherent noise and sparsity of web-crawled data hinder effective pretraining, while multifaceted dermatological knowledge is challenging to exploit from long, unstructured descriptions for pretraining optimization. To address these challenges, we propose a dermatological VLP framework integrating a Multi-Agent data GENeration (MAGEN) system and Ontology-based Multi-Aspect Knowledge-Enhanced (O-MAKE) pretraining. First, MAGEN enhances data quality by synthesizing knowledge-enriched descriptions via a foundation model-assisted captioning and retrieval-based verification pipeline. Second, O-MAKE addresses the difficulty of learning from long, unstructured texts by decomposing them into distinct knowledge aspects. This facilitates fine-grained alignment at both global and patch levels, while explicitly modeling medical concept relationships through ontology-guided mechanisms. Comprehensive experiments demonstrate the effectiveness of each component, achieving state-of-the-art zero-shot performance on disease classification and cross-modal retrieval tasks across eight dermatological datasets. Our code and the augmented dataset Derm1M-AgentAug, comprising over 400K skin image-text pairs, are publicly available at https://github.com/XiejiLi/MAGEN-O-MAKE.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- National Health and Medical Research Council (NHMRC) Centre of Research Excellence in Skin Imaging and Precision Diagnosis
- National Health and Medical Research Council (NHMRC) Synergy Grant: Roadmap Options for Melanoma Screening in Australia
- NHMRC Investigator
Zitieren
APA
Li, X., Yan, S., Liu, Y., Soyer, H. P., Janda, M., Mar, V., & Ge, Z. (2026). Multi-Aspect Knowledge-Enhanced Dermatological Vision–Language Pretraining With Multi-Agent Data Generation. IEEE Transactions on Medical Imaging, 45(11), 5441-5454. https://doi.org/10.1109/tmi.2026.3723004
Vancouver
Li X, Yan S, Liu Y, Soyer HP, Janda M, Mar V, et al. Multi-Aspect Knowledge-Enhanced Dermatological Vision–Language Pretraining With Multi-Agent Data Generation. IEEE Transactions on Medical Imaging. 2026;45(11):5441-5454. doi:10.1109/tmi.2026.3723004
BibTeX
@article{Li2026,
author = {Li, Xieji and Yan, Siyuan and Liu, Yingsheng and Soyer, H Peter and Janda, Monika and Mar, Victoria and Ge, Zongyuan},
title = {Multi-Aspect Knowledge-Enhanced Dermatological Vision–Language Pretraining With Multi-Agent Data Generation},
journal = {IEEE Transactions on Medical Imaging},
year = {2026},
volume = {45},
number = {11},
pages = {5441--5454},
doi = {10.1109/tmi.2026.3723004}
}
EUVIMED