Tracking and Segmenting Anything in Any Modality
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (15) · 12762-12770
DOI: 10.1609/aaai.v40i15.38273 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Tianlu Zhang und 3 weiteren Autor:innen.
Abstract (Originalsprache)
Tracking and segmentation play essential roles in video understanding, providing basic positional information and temporal association of objects within video sequences. Despite their shared objective, existing approaches often tackle these tasks using specialized architectures or modality-specific parameters, limiting their generalization and scalability. Recent efforts have attempted to unify multiple tracking and segmentation sub-tasks from the perspectives of any modality input or multi-task inference. However, these approaches tend to overlook two critical challenges: the distributional gap across different modalities and the feature representation gap across tasks. These issues hinder effective cross-task and cross-modal knowledge sharing, ultimately constraining the development of a true generalist model. To address these limitations, we propose a universal tracking and segmentation framework named SATA, which unifies a broad spectrum of tracking and segmentation subtasks with any modality input. Specifically, a Decoupled Mixture-of-Expert (DeMoE) mechanism is presented to decouple the unified representation learning task into the modeling process of cross-modal shared knowledge and specific information, thus enabling the model to maintain flexibility while enhancing generalization. Additionally, we introduce a Task-aware Multi-object Tracking (TaMOT) pipeline to unify all the task outputs as a unified set of instances with calibrated ID information, thereby alleviating the degradation of task-specific knowledge during multi-task training. SATA demonstrates superior performance on 18 challenging tracking and segmentation benchmarks, offering a novel perspective for more generalizable video understanding.
Zitieren
APA
Zhang, T., Zhang, Q., Ding, G., & Han, J. (2026). Tracking and Segmenting Anything in Any Modality. Proceedings of the AAAI Conference on Artificial Intelligence, 40(15), 12762-12770. https://doi.org/10.1609/aaai.v40i15.38273
Vancouver
Zhang T, Zhang Q, Ding G, Han J. Tracking and Segmenting Anything in Any Modality. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(15):12762-12770. doi:10.1609/aaai.v40i15.38273
BibTeX
@article{Zhang2026,
author = {Zhang, Tianlu and Zhang, Qiang and Ding, Guiguang and Han, Jungong},
title = {Tracking and Segmenting Anything in Any Modality},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {15},
pages = {12762--12770},
doi = {10.1609/aaai.v40i15.38273}
}
EUVIMED