EUVIMEDEUVIMED entdecken
Crossref

Implement Referring Expression Comprehension by Extending Auto-focus Lens to Locked Vision Model

Shiyi Zheng, Peizhi Zhao, Qingbao Huang, Yi Cai, Haonan Cheng, Qi Wu

ACM Transactions on Multimedia Computing, Communications, and Applications · 28.02.2026 · 22 (2) · 1-24

DOI: 10.1145/3777449 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 10.02.2026. Verfasst von Shiyi Zheng und 5 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Guangxi Natural Science Foundation, Guangxi Science and Technology Base and Talent Special.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Referring Expression Comprehension (REC) aims to achieve fine-grained cross-modal content alignment. The traditional two-stage approaches, by decomposing REC into localization (region proposal) and comprehension (expression-based ranking), lead to the isolation of continuous image information and heavily rely on the quality of the proposals. In this article, we propose a point-based two-stage framework for REC to quickly achieve localization by inserting a language-modulated auto-focus module into the locked vision model. Specifically, we redefine REC as two processes: point-based cross-modal comprehension and point-based instance localization. For the comprehension stage, we reconstruct the raw annotations into soft masks at the feature point level as a metric of cross-modal correlation. With this indirect metric, REC can be approximated as a binary classification problem, which fundamentally avoids the impact of isolated regions. Remarkably, soft masks are shape-independent, which means our method is extremely general. By switching different vision models, different types of predictions (e.g., localization and segmentation) can be obtained. Experiments on multiple benchmarks demonstrate the feasibility and potential of our point-based paradigm. Our code will be public at https://github.com/VILAN-Lab/PBREC-AF .

Förderung

  • National Natural Science Foundation of China
  • Guangxi Natural Science Foundation
  • Guangxi Science and Technology Base and Talent Special
  • Communication University of China

Zitieren

APA

Zheng, S., Zhao, P., Huang, Q., Cai, Y., Cheng, H., & Wu, Q. (2026). Implement Referring Expression Comprehension by Extending Auto-focus Lens to Locked Vision Model. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(2), 1-24. https://doi.org/10.1145/3777449

Vancouver

Zheng S, Zhao P, Huang Q, Cai Y, Cheng H, Wu Q. Implement Referring Expression Comprehension by Extending Auto-focus Lens to Locked Vision Model. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(2):1-24. doi:10.1145/3777449

BibTeX
@article{Zheng2026,
  author = {Zheng, Shiyi and Zhao, Peizhi and Huang, Qingbao and Cai, Yi and Cheng, Haonan and Wu, Qi},
  title = {Implement Referring Expression Comprehension by Extending Auto-focus Lens to Locked Vision Model},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2026},
  volume = {22},
  number = {2},
  pages = {1--24},
  doi = {10.1145/3777449}
}
Kostenlos starten →