On-the-Fly Modulation for Balanced Multimodal Learning
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-01 · 47 (1) · 469-485
DOI: 10.1109/tpami.2024.3468315 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-01. Verfasst von Yake Wei und 3 weiteren Autor:innen. Die Arbeit wurde bisher 39-mal zitiert (Crossref) und nennt 61 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Young Elite Scientists Sponsorship Program by CAST, Public Computing Cloud, Renmin University of China.
Abstract (Originalsprache)
Multimodal learning is expected to boost model performance by integrating information from different modalities. However, its potential is not fully exploited because the widely-used joint training strategy, which has a uniform objective for all modalities, leads to imbalanced and under-optimized uni-modal representations. Specifically, we point out that there often exists modality with more discriminative information, e.g., vision of playing football and sound of blowing wind. They could dominate the joint training process, resulting in other modalities being significantly under-optimized. To alleviate this problem, we first analyze the under-optimized phenomenon from both the feed-forward and the back-propagation stages during optimization. Then, On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies are proposed to modulate the optimization of each modality, by monitoring the discriminative discrepancy between modalities during training. Concretely, OPM weakens the influence of the dominant modality by dropping its feature with dynamical probability in the feed-forward stage, while OGM mitigates its gradient in the back-propagation stage. In experiments, our methods demonstrate considerable improvement across a variety of multimodal tasks. These simple yet effective strategies not only enhance performance in vanilla and task-oriented multimodal models, but also in more complex multimodal tasks, showcasing their effectiveness and flexibility.
Förderung
- National Natural Science Foundation of China
- Young Elite Scientists Sponsorship Program by CAST
- Public Computing Cloud, Renmin University of China
Zitieren
APA
Wei, Y., Hu, D., Du, H., & Wen, J. R. (2025). On-the-Fly Modulation for Balanced Multimodal Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(1), 469-485. https://doi.org/10.1109/tpami.2024.3468315
Vancouver
Wei Y, Hu D, Du H, Wen JR. On-the-Fly Modulation for Balanced Multimodal Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(1):469-485. doi:10.1109/tpami.2024.3468315
BibTeX
@article{Wei2025,
author = {Wei, Yake and Hu, Di and Du, Henghui and Wen, Ji-Rong},
title = {On-the-Fly Modulation for Balanced Multimodal Learning},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {1},
pages = {469--485},
doi = {10.1109/tpami.2024.3468315}
}
EUVIMED