EUVIMEDEUVIMED entdecken
Crossref

Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies

Haoran Xu, Peixi Peng, Canming Xia, Guang Tan, Yiqian Chang, Xinyi Zhang, Luntong Li, Yonghong Tian

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026

DOI: 10.1109/tpami.2026.3694266 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. Verfasst von Haoran Xu und 7 weiteren Autor:innen. Hauptthemen laut MEDLINE: Image Processing, Computer-Assisted. Die Arbeit nennt 68 Literaturangaben. Gefördert von: Shenzhen Science and Technology Program, National Natural Science Foundation of China, Major Key Project of Pengcheng Laboratory.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multi-modality visuomotor policies, typically driven by complementary visual sensors such as frame-based RGB cameras and event-based Dynamic Vision Sensors (DVS), have demonstrated strong potential in adverse illumination and high-speed scenarios. Most existing approaches either adopt conventional computer-vision-style feature-level fusion to derive a unified intermediate representation for subsequent value estimation and policy learning, or estimate modality importance separately from a value-level perspective. However, they still face challenges in effectively removing modality-specific task-irrelevant information, leading to suboptimal decisions based on noisy representations. To address this issue, we propose a Decomposed Multi-Modality Representation (DMR) framework that explicitly learns combined task-relevant features (co-features) while separating RGB- and DVS-specific task-irrelevant components. The co-features are trained under task-relevant supervision, while modality-specific components are modeled through reconstruction and InfoNCE-style contrastive objectives to ensure information completeness and discriminability. In addition, existing exploration strategies in reinforcement learning typically encourage transition instability within each modality independently, which often leads to inefficient exploration with limited state-space coverage and low sample efficiency. Therefore, we introduce an intrinsic reward based on Modality Transition Discrepancy (MTD) computed on DMR representations. By quantifying discrepancies between RGB and DVS transition dynamics, MTD promotes exploration of states with meaningful cross-modality differences. Extensive experiments on the CARLA platform and the DDD20 benchmark demonstrate that our method achieves state-of-the-art performance across diverse traffic conditions in both reinforcement learning and imitation learning settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Image Processing, Computer-AssistedHumansReinforcement Machine LearningAlgorithms

Förderung

  • Shenzhen Science and Technology Program
  • National Natural Science Foundation of China
  • Major Key Project of Pengcheng Laboratory

Zitieren

APA

Xu, H., Peng, P., Xia, C., Tan, G., Chang, Y., Zhang, X., Li, L., & Tian, Y. (2026). Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2026.3694266

Vancouver

Xu H, Peng P, Xia C, Tan G, Chang Y, Zhang X, et al. Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026. doi:10.1109/tpami.2026.3694266

BibTeX
@article{Xu2026,
  author = {Xu, Haoran and Peng, Peixi and Xia, Canming and Tan, Guang and Chang, Yiqian and Zhang, Xinyi and Li, Luntong and Tian, Yonghong},
  title = {Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  doi = {10.1109/tpami.2026.3694266}
}
Kostenlos starten →