EUVIMEDEUVIMED entdecken
Crossref

Bi-directional Adapter for Multimodal Tracking

Bing Cao, Junliang Guo, Pengfei Zhu, Qinghua Hu

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (2) · 927-935

DOI: 10.1609/aaai.v38i2.27852 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Bing Cao und 3 weiteren Autor:innen. Die Arbeit wurde bisher 155-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Due to the rapid development of computer vision, single-modal (RGB) object tracking has made significant progress in recent years. Considering the limitation of single imaging sensor, multi-modal images (RGB, infrared, etc.) are introduced to compensate for this deficiency for all-weather object tracking in complex environments. However, as acquiring sufficient multi-modal tracking data is hard while the dominant modality changes with the open environment, most existing techniques fail to extract multi-modal complementary information dynamically, yielding unsatisfactory tracking performance. To handle this problem, we propose a novel multi-modal visual prompt tracking model based on a universal bi-directional adapter, cross-prompting multiple modalities mutually. Our model consists of a universal bi-directional adapter and multiple modality-specific transformer encoder branches with sharing parameters. The encoders extract features of each modality separately by using a frozen, pre-trained foundation model. We develop a simple but effective light feature adapter to transfer modality-specific information from one modality to another, performing visual feature prompt fusion in an adaptive manner. With adding fewer (0.32M) trainable parameters, our model achieves superior tracking performance in comparison with both the full fine-tuning methods and the prompt learning-based methods. Our code is available: https://github.com/SparkTempest/BAT.

Zitieren

APA

Cao, B., Guo, J., Zhu, P., & Hu, Q. (2024). Bi-directional Adapter for Multimodal Tracking. Proceedings of the AAAI Conference on Artificial Intelligence, 38(2), 927-935. https://doi.org/10.1609/aaai.v38i2.27852

Vancouver

Cao B, Guo J, Zhu P, Hu Q. Bi-directional Adapter for Multimodal Tracking. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(2):927-935. doi:10.1609/aaai.v38i2.27852

BibTeX
@article{Cao2024,
  author = {Cao, Bing and Guo, Junliang and Zhu, Pengfei and Hu, Qinghua},
  title = {Bi-directional Adapter for Multimodal Tracking},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {2},
  pages = {927--935},
  doi = {10.1609/aaai.v38i2.27852}
}
Kostenlos starten →