Caption Assisted Multimodal Large Language Model for Video Moment Retrieval
IEEE Transactions on Image Processing · 2025 · 34 · 6755-6766
DOI: 10.1109/tip.2025.3620124 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Peiyu Xie und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 47 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Shenzhen Science and Technology Program, Shenzhen Colleges and Universities Stable Support Program.
Abstract (Originalsprache)
Multimodal Large Language Models (MLLMs) have demonstrated significant potential across various multimodal tasks, including retrieval, summarization, and reasoning. However, it remains a substantial challenge for MLLMs to understand and precisely retrieve specific moments from a video, which require fine-grained spatial and temporal understanding of a video. To overcome this, we propose the Caption Assisted MLLM from Coarse to finE (CALCE), a novel two-stage framework designed for enhanced moment retrieval. Our pipeline begins with a first stage where captions extracted from the audio are utilized to assist the MLLM to provide a robust foundation for precise moment retrieval. To efficiently manage memory consumption from this additional data, a clustering algorithm is applied to the sparsely sampled video frames, categorizing them into key frames and non-key frames. The second stage focuses on recalling missed moments and achieving more fine-grained moment boundaries by adopting a higher sampling rate. In this process, predictions from the first stage cast votes for their correlated densely sampled frames, thereby filtering out less relevant frames. By repeating the process of the first stage with these selected frames, CALCE progressively retrieves video moments from coarse to precise. Experiments on QVHighlights and Charades-STA demonstrate the effectiveness of CALCE, which outperforms existing state-of-the-art methods. The code is available at https://github.com/tjhd1475/CALCE.
Förderung
- National Natural Science Foundation of China
- Shenzhen Science and Technology Program
- Shenzhen Colleges and Universities Stable Support Program
Zitieren
APA
Xie, P., Li, J., Lu, G., Xu, Y., & Zhang, D. (2025). Caption Assisted Multimodal Large Language Model for Video Moment Retrieval. IEEE Transactions on Image Processing, 34, 6755-6766. https://doi.org/10.1109/tip.2025.3620124
Vancouver
Xie P, Li J, Lu G, Xu Y, Zhang D. Caption Assisted Multimodal Large Language Model for Video Moment Retrieval. IEEE Transactions on Image Processing. 2025;34:6755-6766. doi:10.1109/tip.2025.3620124
BibTeX
@article{Xie2025,
author = {Xie, Peiyu and Li, Jinxing and Lu, Guangming and Xu, Yong and Zhang, David},
title = {Caption Assisted Multimodal Large Language Model for Video Moment Retrieval},
journal = {IEEE Transactions on Image Processing},
year = {2025},
volume = {34},
pages = {6755--6766},
doi = {10.1109/tip.2025.3620124}
}
EUVIMED