Multi-Level Fusion for Robust RGBT Tracking via Enhanced Thermal Representation
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.10.2024 · 20 (10) · 1-24
DOI: 10.1145/3678176 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 29.10.2024. Verfasst von Zhangyong Tang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 62 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, 111 Project of Ministry of Education of China.
Abstract (Originalsprache)
Due to the limitations of visible (RGB) sensors in challenging scenarios, such as nighttime and foggy environments, the thermal infrared (TIR) modality draws increasing attention as an auxiliary source for robust tracking systems. Currently, the existing methods extract both the RGB and TIR (RGBT) clues in a similar approach, i.e., utilising RGB-pretrained models with or without finetuning, and then aggregate the multi-modal information through a fusion block embedded in a single level. However, the different imaging principles of RGB and TIR data raise questions about the suitability of RGB-pretrained models for thermal data. In this article, it is argued that the modality gap is overlooked, and an alternative training paradigm is proposed for TIR data to ensure consistency between the training and test data, which is achieved by optimising the TIR feature extractor with only TIR data involved. Furthermore, with the goal of making better use of the enhanced thermal representations, a multi-level fusion strategy is inspired by the observation that various fusion strategies at different levels can contribute to a better performance. Specifically, fusion modules at both the feature and decision levels are derived for a comprehensive fusion procedure while the pixel-level fusion strategy is not considered due to the misalignment of multi-modal image pairs. The effectiveness of our method is demonstrated by extensive qualitative and quantitative experiments conducted on several challenging benchmarks. Code will be released at https://github.com/Zhangyong-Tang/MELT .
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- 111 Project of Ministry of Education of China
- UK EPSRC
Zitieren
APA
Tang, Z., Xu, T., Wu, X., & Kittler, J. (2024). Multi-Level Fusion for Robust RGBT Tracking via Enhanced Thermal Representation. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(10), 1-24. https://doi.org/10.1145/3678176
Vancouver
Tang Z, Xu T, Wu X, Kittler J. Multi-Level Fusion for Robust RGBT Tracking via Enhanced Thermal Representation. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(10):1-24. doi:10.1145/3678176
BibTeX
@article{Tang2024,
author = {Tang, Zhangyong and Xu, Tianyang and Wu, Xiao-Jun and Kittler, Josef},
title = {Multi-Level Fusion for Robust RGBT Tracking via Enhanced Thermal Representation},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {10},
pages = {1--24},
doi = {10.1145/3678176}
}
EUVIMED