FedCMPTrans: Federated Fine-Tuning of Cross-Modality Projection Large Models for Medical Domain
ACM Transactions on Multimedia Computing, Communications, and Applications · 21.11.2026
DOI: 10.1145/3860321 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 21.11.2026. Verfasst von Yuhan Qiu und 8 weiteren Autor:innen. Die Arbeit nennt 44 Literaturangaben.
Abstract (Originalsprache)
Utilizing deep learning models in the medical domain could contribute greatly to healthcare and further applications. For visual tasks such as medical image segmentation, the ability to understand the image and extract vital information is very important. Therefore, large models that show great visual ability in the general domain are introduced. To better harness those pretrained large models in the medical domain, many researchers choose to adapt large models to medical downstream tasks by fine-tuning. However, compared to general domain data, medical data may be harder to acquire and have privacy problems. To this end, Federated Learning (FL) is introduced to utilize distributed data and meanwhile meet the privacy demands. To be specific, we proposed FedCMPTrans, a framework and method to federatedly fine-tune pretrained large models to the medical domain, mainly medical Visual Question Answering (VQA) task. We choose a cross-modality projection model architecture to maintain the ability to generate text, and thus could directly carry out VQA task instead of convert it into classification task. Following previous work, we fine-tune the cross-modality projector model which bridges the visual and text modalities. Additionally, to better utilize the ViT model, a learnable visual prompt is used. Furthermore, to cope with the influence of non-IID data in the FL scenario, we use a contrastive loss term to help with the optimization. Experiments show that FedCMPTrans could acquire good performance on medical VQA tasks while maintaining text generative workflow.
Zitieren
APA
Qiu, Y., Shi, H., Yu, Q., Di, Y., Ma, R., Zhong, X., Song, Z., Liu, Y., & Guan, H. (2026). FedCMPTrans: Federated Fine-Tuning of Cross-Modality Projection Large Models for Medical Domain. ACM Transactions on Multimedia Computing, Communications, and Applications. https://doi.org/10.1145/3860321
Vancouver
Qiu Y, Shi H, Yu Q, Di Y, Ma R, Zhong X, et al. FedCMPTrans: Federated Fine-Tuning of Cross-Modality Projection Large Models for Medical Domain. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026. doi:10.1145/3860321
BibTeX
@article{Qiu2026,
author = {Qiu, Yuhan and Shi, Hongjian and Yu, Qiwei and Di, Yicheng and Ma, Ruhui and Zhong, Xiaohui and Song, Zhiwei and Liu, Yuan and Guan, Haibing},
title = {FedCMPTrans: Federated Fine-Tuning of Cross-Modality Projection Large Models for Medical Domain},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2026},
doi = {10.1145/3860321}
}
EUVIMED