Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026
DOI: 10.1109/tpami.2026.3694266 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. Verfasst von Haoran Xu und 7 weiteren Autor:innen. Hauptthemen laut MEDLINE: Image Processing, Computer-Assisted. Die Arbeit nennt 68 Literaturangaben. Gefördert von: Shenzhen Science and Technology Program, National Natural Science Foundation of China, Major Key Project of Pengcheng Laboratory.
Abstract (Originalsprache)
Multi-modality visuomotor policies, typically driven by complementary visual sensors such as frame-based RGB cameras and event-based Dynamic Vision Sensors (DVS), have demonstrated strong potential in adverse illumination and high-speed scenarios. Most existing approaches either adopt conventional computer-vision-style feature-level fusion to derive a unified intermediate representation for subsequent value estimation and policy learning, or estimate modality importance separately from a value-level perspective. However, they still face challenges in effectively removing modality-specific task-irrelevant information, leading to suboptimal decisions based on noisy representations. To address this issue, we propose a Decomposed Multi-Modality Representation (DMR) framework that explicitly learns combined task-relevant features (co-features) while separating RGB- and DVS-specific task-irrelevant components. The co-features are trained under task-relevant supervision, while modality-specific components are modeled through reconstruction and InfoNCE-style contrastive objectives to ensure information completeness and discriminability. In addition, existing exploration strategies in reinforcement learning typically encourage transition instability within each modality independently, which often leads to inefficient exploration with limited state-space coverage and low sample efficiency. Therefore, we introduce an intrinsic reward based on Modality Transition Discrepancy (MTD) computed on DMR representations. By quantifying discrepancies between RGB and DVS transition dynamics, MTD promotes exploration of states with meaningful cross-modality differences. Extensive experiments on the CARLA platform and the DDD20 benchmark demonstrate that our method achieves state-of-the-art performance across diverse traffic conditions in both reinforcement learning and imitation learning settings.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- Shenzhen Science and Technology Program
- National Natural Science Foundation of China
- Major Key Project of Pengcheng Laboratory
Zitieren
APA
Xu, H., Peng, P., Xia, C., Tan, G., Chang, Y., Zhang, X., Li, L., & Tian, Y. (2026). Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2026.3694266
Vancouver
Xu H, Peng P, Xia C, Tan G, Chang Y, Zhang X, et al. Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026. doi:10.1109/tpami.2026.3694266
BibTeX
@article{Xu2026,
author = {Xu, Haoran and Peng, Peixi and Xia, Canming and Tan, Guang and Chang, Yiqian and Zhang, Xinyi and Li, Luntong and Tian, Yonghong},
title = {Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
doi = {10.1109/tpami.2026.3694266}
}
EUVIMED