Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution
IEEE Transactions on Image Processing · 2025 · 34 · 7235-7249
DOI: 10.1109/tip.2025.3611687 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Zhangyong Tang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 69 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, 111 Project of Ministry of Education of China.
Abstract (Originalsprache)
RGBT tracking draws increasing attention because of its robustness in multi-modal warranting (MMW) scenarios, such as nighttime and adverse weather conditions, where relying on a single sensing modality fails to ensure stable tracking results. However, existing benchmarks predominantly contain videos collected in common scenarios where both RGB and thermal infrared (TIR) information are of sufficient quality. This weakens the representativeness of existing benchmarks in severe imaging conditions, leading to tracking failures in MMW scenarios. To bridge this gap, we present a new benchmark considering the modality validity, MV-RGBT, captured specifically from MMW scenarios where either RGB (extreme illumination) or TIR (thermal truncation) modality is invalid. Hence, it is further divided into two subsets according to the valid modality, offering a new compositional perspective for evaluation and providing valuable insights for future designs. Moreover, MV-RGBT is the most diverse benchmark of its kind, featuring 36 different object categories captured across 19 distinct scenes. Furthermore, considering severe imaging conditions in MMW scenarios, a new problem is posed in RGBT tracking, named 'when to fuse', to stimulate the development of fusion strategies for such scenarios. To facilitate its discussion, we propose a new solution with a mixture of experts, named MoETrack, where each expert generates independent tracking results along with a confidence score. Extensive results demonstrate the significant potential of MV-RGBT in advancing RGBT tracking and elicit the conclusion that fusion is not always beneficial, especially in MMW scenarios. Besides, MoETrack achieves state-of-the-art results on several benchmarks, including MV-RGBT, GTOT, and LasHeR. Source codes and benchmarks are available at https://github.com/Zhangyong-Tang/MVRGBT.
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- 111 Project of Ministry of Education of China
- Fundamental Research Funds for the Central Universities
- Wuxi Science and Technology Development Fund Project
- Leverhulme Trust Emeritus Fellowship
Zitieren
APA
Tang, Z., Xu, T., Wu, X. J., Zhu, X., Cheng, C., Feng, Z., & Kittler, J. (2025). Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution. IEEE Transactions on Image Processing, 34, 7235-7249. https://doi.org/10.1109/tip.2025.3611687
Vancouver
Tang Z, Xu T, Wu XJ, Zhu X, Cheng C, Feng Z, et al. Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution. IEEE Transactions on Image Processing. 2025;34:7235-7249. doi:10.1109/tip.2025.3611687
BibTeX
@article{Tang2025,
author = {Tang, Zhangyong and Xu, Tianyang and Wu, Xiao-Jun and Zhu, Xuefeng and Cheng, Chunyang and Feng, Zhenhua and Kittler, Josef},
title = {Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution},
journal = {IEEE Transactions on Image Processing},
year = {2025},
volume = {34},
pages = {7235--7249},
doi = {10.1109/tip.2025.3611687}
}
EUVIMED