EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal Features Alignment for Vision–Language Object Tracking

Ping Ye, Gang Xiao, Jun Liu

Remote Sensing · 27.03.2024 · 16 (7) · 1168

DOI: 10.3390/rs16071168 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Remote Sensing, online erschienen am 27.03.2024. Verfasst von Ping Ye und 2 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Artificial Intelligence Key Laboratory of Sichuan Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Vision–language tracking presents a crucial challenge in multimodal object tracking. Integrating language features and visual features can enhance target localization and improve the stability and accuracy of the tracking process. However, most existing fusion models in vision–language trackers simply concatenate visual and linguistic features without considering their semantic relationships. Such methods fail to distinguish the target’s appearance features from the background, particularly when the target changes dramatically. To address these limitations, we introduce an innovative technique known as multimodal features alignment (MFA) for vision–language tracking. In contrast to basic concatenation methods, our approach employs a factorized bilinear pooling method that conducts squeezing and expanding operations to create a unified feature representation from visual and linguistic features. Moreover, we integrate the co-attention mechanism twice to derive varied weights for the search region, ensuring that higher weights are placed on the aligned visual and linguistic features. Subsequently, the fused feature map with diverse distributed weights serves as the search region during the tracking phase, facilitating anchor-free grounding to predict the target’s location. Extensive experiments are conducted on multiple public datasets, and our proposed tracker obtains a success score of 0.654/0.553/0.447 and a precision score of 0.872/0.556/0.513 on OTB-LANG/LaSOT/TNL2K. These results are satisfying compared with those of recent state-of-the-art vision–language trackers.

Förderung

  • National Natural Science Foundation of China
  • Artificial Intelligence Key Laboratory of Sichuan Province

Zitieren

APA

Ye, P., Xiao, G., & Liu, J. (2024). Multimodal Features Alignment for Vision–Language Object Tracking. Remote Sensing, 16(7), 1168. https://doi.org/10.3390/rs16071168

Vancouver

Ye P, Xiao G, Liu J. Multimodal Features Alignment for Vision–Language Object Tracking. Remote Sensing. 2024;16(7):1168. doi:10.3390/rs16071168

BibTeX
@article{Ye2024,
  author = {Ye, Ping and Xiao, Gang and Liu, Jun},
  title = {Multimodal Features Alignment for Vision–Language Object Tracking},
  journal = {Remote Sensing},
  year = {2024},
  volume = {16},
  number = {7},
  pages = {1168},
  doi = {10.3390/rs16071168}
}
Kostenlos starten →