EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking

Jin Shi, Huixin Zhou, Shuxia Qi, Xiang Teng, Jiangluqi Song, Pei Xiang, Soon Ki Jung

Remote Sensing · 20.11.2026 · 18 (22) · 4004

DOI: 10.3390/rs18224004 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Remote Sensing, online erschienen am 20.11.2026. Verfasst von Jin Shi und 6 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Science Basic Research Program of Shaanxi Province under Grant, National Natural Science Foundation of China, China Postdoctoral Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

RGB-thermal infrared (RGBT) target tracking integrates spatial features of both RGB and thermal infrared (TIR) modalities, providing a discriminative basis for target feature representation and state estimation. However, existing methods suffer from insufficient cross-modal feature interaction and inadequate spatio-temporal contextual correlation, which limits tracking stability and accuracy in challenging scenarios. To address these issues, we propose MSTrack, a novel tracker that integrates a multi-modal feature aggregation (MMFA) module and a spatio-temporal memory context encoding (STMCE) module. First, to capture both shared features and complementary details across modalities, we design the MMFA module, which employs parallel Swin Transformers and four cross-modal feature interaction (CMFI) components to achieve progressive fusion of RGB and TIR features. Specifically, CMFI integrates a cross-modal collaborative attention mechanism, a triplet attention mechanism, a bidirectional cross-modal weighted fusion mechanism, and a multi-stage information aggregation mechanism to achieve cross-modal interactions. Second, to fuse the key spatio-temporal features across multiple frames, we design the STMCE module. This STMCE module leverages a historical feature memory pool (HFMP) and a spatio-temporal context encoder (STCE) to model global dependencies among the template, current, and historical frames. Finally, the prediction head outputs the target bounding box. Experiments on four ground-based RGBT datasets demonstrate that the proposed tracker outperforms other state-of-the-art trackers. Furthermore, the results on the UAV-based VTUAV dataset validate its generalization potential. Specifically, MSTrack achieves PR/SR scores of 0.877/0.637 on RGBT234 and 0.669/0.530 on LasHeR. On the VTUAV dataset, it achieves MPR/MSR scores of 0.812/0.672 on the short-term subset and 0.595/0.501 on the long-term subset.

Förderung

  • Natural Science Basic Research Program of Shaanxi Province under Grant
  • National Natural Science Foundation of China
  • China Postdoctoral Science Foundation
  • 111 Project

Zitieren

APA

Shi, J., Zhou, H., Qi, S., Teng, X., Song, J., Xiang, P., & Jung, S. K. (2026). MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking. Remote Sensing, 18(22), 4004. https://doi.org/10.3390/rs18224004

Vancouver

Shi J, Zhou H, Qi S, Teng X, Song J, Xiang P, et al. MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking. Remote Sensing. 2026;18(22):4004. doi:10.3390/rs18224004

BibTeX
@article{Shi2026,
  author = {Shi, Jin and Zhou, Huixin and Qi, Shuxia and Teng, Xiang and Song, Jiangluqi and Xiang, Pei and Jung, Soon Ki},
  title = {MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking},
  journal = {Remote Sensing},
  year = {2026},
  volume = {18},
  number = {22},
  pages = {4004},
  doi = {10.3390/rs18224004}
}
Kostenlos starten →