MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking
Remote Sensing · 20.11.2026 · 18 (22) · 4004
DOI: 10.3390/rs18224004 ↗Kurz gefasst
Fachartikel in der Zeitschrift Remote Sensing, online erschienen am 20.11.2026. Verfasst von Jin Shi und 6 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Science Basic Research Program of Shaanxi Province under Grant, National Natural Science Foundation of China, China Postdoctoral Science Foundation.
Abstract (Originalsprache)
RGB-thermal infrared (RGBT) target tracking integrates spatial features of both RGB and thermal infrared (TIR) modalities, providing a discriminative basis for target feature representation and state estimation. However, existing methods suffer from insufficient cross-modal feature interaction and inadequate spatio-temporal contextual correlation, which limits tracking stability and accuracy in challenging scenarios. To address these issues, we propose MSTrack, a novel tracker that integrates a multi-modal feature aggregation (MMFA) module and a spatio-temporal memory context encoding (STMCE) module. First, to capture both shared features and complementary details across modalities, we design the MMFA module, which employs parallel Swin Transformers and four cross-modal feature interaction (CMFI) components to achieve progressive fusion of RGB and TIR features. Specifically, CMFI integrates a cross-modal collaborative attention mechanism, a triplet attention mechanism, a bidirectional cross-modal weighted fusion mechanism, and a multi-stage information aggregation mechanism to achieve cross-modal interactions. Second, to fuse the key spatio-temporal features across multiple frames, we design the STMCE module. This STMCE module leverages a historical feature memory pool (HFMP) and a spatio-temporal context encoder (STCE) to model global dependencies among the template, current, and historical frames. Finally, the prediction head outputs the target bounding box. Experiments on four ground-based RGBT datasets demonstrate that the proposed tracker outperforms other state-of-the-art trackers. Furthermore, the results on the UAV-based VTUAV dataset validate its generalization potential. Specifically, MSTrack achieves PR/SR scores of 0.877/0.637 on RGBT234 and 0.669/0.530 on LasHeR. On the VTUAV dataset, it achieves MPR/MSR scores of 0.812/0.672 on the short-term subset and 0.595/0.501 on the long-term subset.
Förderung
- Natural Science Basic Research Program of Shaanxi Province under Grant
- National Natural Science Foundation of China
- China Postdoctoral Science Foundation
- 111 Project
Zitieren
APA
Shi, J., Zhou, H., Qi, S., Teng, X., Song, J., Xiang, P., & Jung, S. K. (2026). MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking. Remote Sensing, 18(22), 4004. https://doi.org/10.3390/rs18224004
Vancouver
Shi J, Zhou H, Qi S, Teng X, Song J, Xiang P, et al. MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking. Remote Sensing. 2026;18(22):4004. doi:10.3390/rs18224004
BibTeX
@article{Shi2026,
author = {Shi, Jin and Zhou, Huixin and Qi, Shuxia and Teng, Xiang and Song, Jiangluqi and Xiang, Pei and Jung, Soon Ki},
title = {MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking},
journal = {Remote Sensing},
year = {2026},
volume = {18},
number = {22},
pages = {4004},
doi = {10.3390/rs18224004}
}
EUVIMED