EUVIMEDEUVIMED entdecken
Crossref

Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution

Zhangyong Tang, Tianyang Xu, Xiao-Jun Wu, Xuefeng Zhu, Chunyang Cheng, Zhenhua Feng, Josef Kittler

IEEE Transactions on Image Processing · 2025 · 34 · 7235-7249

DOI: 10.1109/tip.2025.3611687 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Zhangyong Tang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 69 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, 111 Project of Ministry of Education of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

RGBT tracking draws increasing attention because of its robustness in multi-modal warranting (MMW) scenarios, such as nighttime and adverse weather conditions, where relying on a single sensing modality fails to ensure stable tracking results. However, existing benchmarks predominantly contain videos collected in common scenarios where both RGB and thermal infrared (TIR) information are of sufficient quality. This weakens the representativeness of existing benchmarks in severe imaging conditions, leading to tracking failures in MMW scenarios. To bridge this gap, we present a new benchmark considering the modality validity, MV-RGBT, captured specifically from MMW scenarios where either RGB (extreme illumination) or TIR (thermal truncation) modality is invalid. Hence, it is further divided into two subsets according to the valid modality, offering a new compositional perspective for evaluation and providing valuable insights for future designs. Moreover, MV-RGBT is the most diverse benchmark of its kind, featuring 36 different object categories captured across 19 distinct scenes. Furthermore, considering severe imaging conditions in MMW scenarios, a new problem is posed in RGBT tracking, named 'when to fuse', to stimulate the development of fusion strategies for such scenarios. To facilitate its discussion, we propose a new solution with a mixture of experts, named MoETrack, where each expert generates independent tracking results along with a confidence score. Extensive results demonstrate the significant potential of MV-RGBT in advancing RGBT tracking and elicit the conclusion that fusion is not always beneficial, especially in MMW scenarios. Besides, MoETrack achieves state-of-the-art results on several benchmarks, including MV-RGBT, GTOT, and LasHeR. Source codes and benchmarks are available at https://github.com/Zhangyong-Tang/MVRGBT.

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • 111 Project of Ministry of Education of China
  • Fundamental Research Funds for the Central Universities
  • Wuxi Science and Technology Development Fund Project
  • Leverhulme Trust Emeritus Fellowship

Zitieren

APA

Tang, Z., Xu, T., Wu, X. J., Zhu, X., Cheng, C., Feng, Z., & Kittler, J. (2025). Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution. IEEE Transactions on Image Processing, 34, 7235-7249. https://doi.org/10.1109/tip.2025.3611687

Vancouver

Tang Z, Xu T, Wu XJ, Zhu X, Cheng C, Feng Z, et al. Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution. IEEE Transactions on Image Processing. 2025;34:7235-7249. doi:10.1109/tip.2025.3611687

BibTeX
@article{Tang2025,
  author = {Tang, Zhangyong and Xu, Tianyang and Wu, Xiao-Jun and Zhu, Xuefeng and Cheng, Chunyang and Feng, Zhenhua and Kittler, Josef},
  title = {Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution},
  journal = {IEEE Transactions on Image Processing},
  year = {2025},
  volume = {34},
  pages = {7235--7249},
  doi = {10.1109/tip.2025.3611687}
}
Kostenlos starten →