Reconcile Gradient Modulation for Harmony Multimodal Learning
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (25) · 21225-21233
DOI: 10.1609/aaai.v40i25.39267 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xiyuan Gao und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Multimodal learning frequently faces two coupled challenges: modality imbalance, where dominant modalities suppress others during training, and modality conflict, where opposing gradient directions hinder optimization. Existing methods typically address these issues in isolation, yet they are intrinsically correlated and most fundamentally reflected in the gradient space—severe imbalance may obscure conflicts, while suppressing conflict may homogenize features and worsen imbalance, affecting fusion performance. To jointly address this coupled challenge, we propose Reconcile Gradient Modulation (RGM), a unified framework that adaptively adjusts gradient magnitude and direction for harmony multimodal learning. The core of RGM is SynOrth Grad, which minimizes Dirichlet energy to perform minimal-gradient surgery. It enhances cooperation synergy when modalities are aligned and enforces orthogonality to preserve uniqueness in conflict situations, thus promoting stable and balanced learning. To guide this modulation, we propose Cumulative Gradient Energy (CGE) as a convergence-guaranteed measure of modality-wise progress, and construct a Balance-nonConflict Plane (BCP) for real-time diagnosis and control of training dynamics. Experiments on diverse benchmarks validate our effectiveness and generalizability, consistently outperforming counterparts that are designed to handle multimodal imbalance or conflict independently.
Zitieren
APA
Gao, X., Cao, B., Gong, B., & Zhu, P. (2026). Reconcile Gradient Modulation for Harmony Multimodal Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(25), 21225-21233. https://doi.org/10.1609/aaai.v40i25.39267
Vancouver
Gao X, Cao B, Gong B, Zhu P. Reconcile Gradient Modulation for Harmony Multimodal Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(25):21225-21233. doi:10.1609/aaai.v40i25.39267
BibTeX
@article{Gao2026,
author = {Gao, Xiyuan and Cao, Bing and Gong, Baoquan and Zhu, Pengfei},
title = {Reconcile Gradient Modulation for Harmony Multimodal Learning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {25},
pages = {21225--21233},
doi = {10.1609/aaai.v40i25.39267}
}
EUVIMED