Sa2VA: Marrying SAM2 With MLLM for Dense Grounded Understanding of Images and Videos
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-11 · 48 (11) · 14922-14936
DOI: 10.1109/tpami.2026.3720379 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-11. Verfasst von Haobo Yuan und 10 weiteren Autor:innen. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 125 Literaturangaben. Gefördert von: Intelligence Advanced Research Projects Activity, Institute of Information & Communications Technology Planning & Evaluation, Korean Government.
Abstract (Originalsprache)
This work presents Sa2VA, the first comprehensive, unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models, which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal single-stage instruction tuning. Sa2VA combines SAM-2, a foundation video segmentation model, with MLLM, advanced vision-language models, and unifies text, image, and video into a shared LLM token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2 in producing precise masks, enabling a grounded, multi-modal understanding of both static and dynamic visual content. Additionally, we introduce Ref-SAV, an auto-labeled dataset containing over 72 k object expressions in complex video scenes, designed to boost model performance. We also manually validate 2 k video objects in the Ref-SAV datasets to benchmark referring video object segmentation in complex environments. Experiments show that Sa2VA achieves strong performance across multiple tasks, particularly in referring video object segmentation, highlighting its potential for complex real-world applications. In addition, Sa2VA can be easily extended into various MLLMs, including Qwen-VL and Intern-VL, which can be updated with rapid process in current open-sourced MLLMs. Code and models have been provided to the community.
Förderung
- Intelligence Advanced Research Projects Activity
- Institute of Information & Communications Technology Planning & Evaluation
- Korean Government
- National AI Research Lab Project
- National Natural Science Foundation of China
Zitieren
APA
Yuan, H., Li, X., Zhang, T., Sun, Y., Huang, Z., Xu, S., Ji, S., Tong, Y., Qi, L., Feng, J., & Yang, M. H. (2026). Sa2VA: Marrying SAM2 With MLLM for Dense Grounded Understanding of Images and Videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(11), 14922-14936. https://doi.org/10.1109/tpami.2026.3720379
Vancouver
Yuan H, Li X, Zhang T, Sun Y, Huang Z, Xu S, et al. Sa2VA: Marrying SAM2 With MLLM for Dense Grounded Understanding of Images and Videos. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(11):14922-14936. doi:10.1109/tpami.2026.3720379
BibTeX
@article{Yuan2026,
author = {Yuan, Haobo and Li, Xiangtai and Zhang, Tao and Sun, Yueyi and Huang, Zilong and Xu, Shilin and Ji, Shunping and Tong, Yunhai and Qi, Lu and Feng, Jiashi and Yang, Ming-Hsuan},
title = {Sa2VA: Marrying SAM2 With MLLM for Dense Grounded Understanding of Images and Videos},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
volume = {48},
number = {11},
pages = {14922--14936},
doi = {10.1109/tpami.2026.3720379}
}
EUVIMED