EUVIMEDEUVIMED entdecken
Crossref

JointFormer: A Unified Framework With Joint Modeling for Video Object Segmentation

Jiaming Zhang, Yutao Cui, Gangshan Wu, Limin Wang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-07 · 47 (7) · 6039-6054

DOI: 10.1109/tpami.2025.3557841 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-07. Verfasst von Jiaming Zhang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 82 Literaturangaben. Gefördert von: National Key R&D Program of China, Fundamental Research Funds for the Central Universities, Jiangsu Frontier Technology Research and Development.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Current prevailing Video Object Segmentation (VOS) methods follow the pipeline of extraction-then-matching, which first extracts features on current and reference frames independently, and then performs dense matching between them. This decoupled pipeline limits information propagation between frames to high-level features, and fails to capture fine-grained details for matching. Furthermore, the pixel-wise matching lacks holistic target understanding, making it prone to disturbance by similar distractors. To address these issues, we propose a unified VOS framework, coined JointFormer, for jointly modeling feature extraction, correspondence matching, and a compressed memory. The Joint Modeling Block leverages attention operations to simultaneously extract and propagate the target information from the reference frame to the current frame and a compressed memory token.This joint modeling scheme enables extensive multi-layer propagation beyond high-level feature space and facilitates robust instance-distinctive feature learning. In addition, to incorporate the long-term and holistic target information, we introduce a compressed memory token with a customized online updating mechanism, which aggregates target features and performs temporal information propagation in a frame-wise manner, enhancing the global modeling consistency. Our JointFormer achieves a new state-of-the-art performance on the DAVIS 2017 val/test-dev (89.7% and 87.6%) benchmarks and the YouTube-VOS 2018/2019 val (87.0% and 87.0%) benchmarks. To demonstrate the generalizability of JointFormer, it is further evaluated on four new benchmarks with various challenges, including MOSE for complex scenes, VISOR for egocentric videos, VOST for complex transformations, and LVOS for long-term videos. Without specific design to address these unusual difficulties, our model achieves the best performance across all benchmarks when compared with several current best models, illustrating its excellent generalization and robustness. Further extensive ablations and visualizations indicate our JointFormer enables more comprehensive and effective feature learning and matching.

Förderung

  • National Key R&D Program of China
  • Fundamental Research Funds for the Central Universities
  • Jiangsu Frontier Technology Research and Development
  • Collaborative Innovation Center of Novel Software Technology and Industrialization

Zitieren

APA

Zhang, J., Cui, Y., Wu, G., & Wang, L. (2025). JointFormer: A Unified Framework With Joint Modeling for Video Object Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(7), 6039-6054. https://doi.org/10.1109/tpami.2025.3557841

Vancouver

Zhang J, Cui Y, Wu G, Wang L. JointFormer: A Unified Framework With Joint Modeling for Video Object Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(7):6039-6054. doi:10.1109/tpami.2025.3557841

BibTeX
@article{Zhang2025,
  author = {Zhang, Jiaming and Cui, Yutao and Wu, Gangshan and Wang, Limin},
  title = {JointFormer: A Unified Framework With Joint Modeling for Video Object Segmentation},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {7},
  pages = {6039--6054},
  doi = {10.1109/tpami.2025.3557841}
}
Kostenlos starten →