EUVIMEDEUVIMED entdecken
Crossref

Caption Assisted Multimodal Large Language Model for Video Moment Retrieval

Peiyu Xie, Jinxing Li, Guangming Lu, Yong Xu, David Zhang

IEEE Transactions on Image Processing · 2025 · 34 · 6755-6766

DOI: 10.1109/tip.2025.3620124 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Peiyu Xie und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 47 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Shenzhen Science and Technology Program, Shenzhen Colleges and Universities Stable Support Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal Large Language Models (MLLMs) have demonstrated significant potential across various multimodal tasks, including retrieval, summarization, and reasoning. However, it remains a substantial challenge for MLLMs to understand and precisely retrieve specific moments from a video, which require fine-grained spatial and temporal understanding of a video. To overcome this, we propose the Caption Assisted MLLM from Coarse to finE (CALCE), a novel two-stage framework designed for enhanced moment retrieval. Our pipeline begins with a first stage where captions extracted from the audio are utilized to assist the MLLM to provide a robust foundation for precise moment retrieval. To efficiently manage memory consumption from this additional data, a clustering algorithm is applied to the sparsely sampled video frames, categorizing them into key frames and non-key frames. The second stage focuses on recalling missed moments and achieving more fine-grained moment boundaries by adopting a higher sampling rate. In this process, predictions from the first stage cast votes for their correlated densely sampled frames, thereby filtering out less relevant frames. By repeating the process of the first stage with these selected frames, CALCE progressively retrieves video moments from coarse to precise. Experiments on QVHighlights and Charades-STA demonstrate the effectiveness of CALCE, which outperforms existing state-of-the-art methods. The code is available at https://github.com/tjhd1475/CALCE.

Förderung

  • National Natural Science Foundation of China
  • Shenzhen Science and Technology Program
  • Shenzhen Colleges and Universities Stable Support Program

Zitieren

APA

Xie, P., Li, J., Lu, G., Xu, Y., & Zhang, D. (2025). Caption Assisted Multimodal Large Language Model for Video Moment Retrieval. IEEE Transactions on Image Processing, 34, 6755-6766. https://doi.org/10.1109/tip.2025.3620124

Vancouver

Xie P, Li J, Lu G, Xu Y, Zhang D. Caption Assisted Multimodal Large Language Model for Video Moment Retrieval. IEEE Transactions on Image Processing. 2025;34:6755-6766. doi:10.1109/tip.2025.3620124

BibTeX
@article{Xie2025,
  author = {Xie, Peiyu and Li, Jinxing and Lu, Guangming and Xu, Yong and Zhang, David},
  title = {Caption Assisted Multimodal Large Language Model for Video Moment Retrieval},
  journal = {IEEE Transactions on Image Processing},
  year = {2025},
  volume = {34},
  pages = {6755--6766},
  doi = {10.1109/tip.2025.3620124}
}
Kostenlos starten →