RemixFormer++: A Multi-Modal Transformer Model for Precision Skin Tumor Differential Diagnosis With Memory-Efficient Attention
IEEE Transactions on Medical Imaging · 2025
DOI: 10.1109/tmi.2024.3441012 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2025. Verfasst von Jing Xu und 14 weiteren Autor:innen. Hauptthemen laut MEDLINE: Skin Neoplasms, Image Interpretation, Computer-Assisted. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 50 Literaturangaben. Gefördert von: National Key Research and Development Program of China, China Postdoctoral Science Foundation, Postdoctoral Fellowship Program.
Abstract (Originalsprache)
Diagnosing malignant skin tumors accurately at an early stage can be challenging due to ambiguous and even confusing visual characteristics displayed by various categories of skin tumors. To improve diagnosis precision, all available clinical data from multiple sources, particularly clinical images, dermoscopy images, and medical history, could be considered. Aligning with clinical practice, we propose a novel Transformer model, named RemixFormer++ that consists of a clinical image branch, a dermoscopy image branch, and a metadata branch. Given the unique characteristics inherent in clinical and dermoscopy images, specialized attention strategies are adopted for each type. Clinical images are processed through a top-down architecture, capturing both localized lesion details and global contextual information. Conversely, dermoscopy images undergo a bottom-up processing with two-level hierarchical encoders, designed to pinpoint fine-grained structural and textural features. A dedicated metadata branch seamlessly integrates non-visual information by encoding relevant patient data. Fusing features from three branches substantially boosts disease classification accuracy. RemixFormer++ demonstrates exceptional performance on four single-modality datasets (PAD-UFES-20, ISIC 2017/2018/2019). Compared with the previous best method using a public multi-modal Derm7pt dataset, we achieved an absolute 5.3% increase in averaged F1 and 1.2% in accuracy for the classification of five skin tumors. Furthermore, using a large-scale in-house dataset of 10,351 patients with the twelve most common skin tumors, our method obtained an overall classification accuracy of 92.6%. These promising results, on par or better with the performance of 191 dermatologists through a comprehensive reader study, evidently imply the potential clinical usability of our method.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- National Key Research and Development Program of China
- China Postdoctoral Science Foundation
- Postdoctoral Fellowship Program
- Hunan Provincial Natural Science Foundation
Zitieren
APA
Xu, J., Huang, K., Zhong, L., Gao, Y., Sun, K., Liu, W., Zhou, Y., Guo, W., Guo, Y., Zou, Y., Duan, Y., Lu, L., Wang, Y., Chen, X., & Zhao, S. (2025). RemixFormer++: A Multi-Modal Transformer Model for Precision Skin Tumor Differential Diagnosis With Memory-Efficient Attention. IEEE Transactions on Medical Imaging. https://doi.org/10.1109/tmi.2024.3441012
Vancouver
Xu J, Huang K, Zhong L, Gao Y, Sun K, Liu W, et al. RemixFormer++: A Multi-Modal Transformer Model for Precision Skin Tumor Differential Diagnosis With Memory-Efficient Attention. IEEE Transactions on Medical Imaging. 2025. doi:10.1109/tmi.2024.3441012
BibTeX
@article{Xu2025,
author = {Xu, Jing and Huang, Kai and Zhong, Lianzhen and Gao, Yuan and Sun, Kai and Liu, Wei and Zhou, Yanjie and Guo, Wenchao and Guo, Yuan and Zou, Yuanqiang and Duan, Yuping and Lu, Le and Wang, Yu and Chen, Xiang and Zhao, Shuang},
title = {RemixFormer++: A Multi-Modal Transformer Model for Precision Skin Tumor Differential Diagnosis With Memory-Efficient Attention},
journal = {IEEE Transactions on Medical Imaging},
year = {2025},
doi = {10.1109/tmi.2024.3441012}
}
EUVIMED