Hybrid Uncertainty Metrics-Based Privacy-Preserving Alternating Multimodal Representation Learning
Applied Sciences · 08.05.2025 · 15 (10) · 5229
DOI: 10.3390/app15105229 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 08.05.2025. Verfasst von Zhe Sun und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key R&D Program of China, Major Research plan of the National Natural Science Foundation of China, National Natural Science Foundation of China.
Abstract (Originalsprache)
Multimodal learning enhances model performance by integrating heterogeneous data but is hindered by modality laziness and privacy vulnerabilities. Modality laziness occurs when the model overly relies on a single modality for predictions, underutilizing other modalities and leading to suboptimal performance and poor cross-modal integration. Privacy vulnerabilities arise when sensitive data from individual modalities are exposed during training or inference, risking unauthorized access or attacks, especially in shared model components. In this paper, we propose Privacy-Preserving Alternating Multimodal Representation Learning (PAMRL). Built on Multimodal Learning with Alternating Unimodal Adaptation (MLA), PAMRL alternately optimizes unimodal encoders and a shared representation head to mitigate modality laziness and improve cross-modal consistency. It introduces a hybrid uncertainty metric combining KL divergence and entropy to enhance prediction robustness while applying differential privacy to protect sensitive data in unimodal encoders, preserving the shared head for efficient cross-modal fusion. Extensive experiments on the MVSA and CREMA-D datasets, comparing PAMRL with MLA and other baselines, demonstrate its superior performance, achieving an optimal balance of predictive accuracy, attack resilience, and privacy protection, thus supporting secure, efficient multimodal applications.
Förderung
- National Key R&D Program of China
- Major Research plan of the National Natural Science Foundation of China
- National Natural Science Foundation of China
- Guangdong Basic and Applied Basic Research Foundation
- Guangzhou Municipal Bureau of Education Higher Education Research Project
Zitieren
APA
Sun, Z., Huang, Y., Zhang, A., Li, C., Jiang, L., Liao, X., Li, R., & Wan, J. (2025). Hybrid Uncertainty Metrics-Based Privacy-Preserving Alternating Multimodal Representation Learning. Applied Sciences, 15(10), 5229. https://doi.org/10.3390/app15105229
Vancouver
Sun Z, Huang Y, Zhang A, Li C, Jiang L, Liao X, et al. Hybrid Uncertainty Metrics-Based Privacy-Preserving Alternating Multimodal Representation Learning. Applied Sciences. 2025;15(10):5229. doi:10.3390/app15105229
BibTeX
@article{Sun2025,
author = {Sun, Zhe and Huang, Yaowei and Zhang, Aohai and Li, Chao and Jiang, Lifan and Liao, Xiaotong and Li, Ran and Wan, Junping},
title = {Hybrid Uncertainty Metrics-Based Privacy-Preserving Alternating Multimodal Representation Learning},
journal = {Applied Sciences},
year = {2025},
volume = {15},
number = {10},
pages = {5229},
doi = {10.3390/app15105229}
}
EUVIMED