CoVR: Learning Composed Video Retrieval from Web Video Captions
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (6) · 5270-5279
DOI: 10.1609/aaai.v38i6.28334 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Lucas Ventura und 3 weiteren Autor:innen. Die Arbeit wurde bisher 56-mal zitiert (Crossref).
Abstract (Originalsprache)
Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together, to search for relevant images in a database. Most CoIR approaches require manually annotated datasets, comprising image-text-image triplets, where the text describes a modification from the query image to the target image. However, manual curation of CoIR triplets is expensive and prevents scalability. In this work, we instead propose a scalable automatic dataset creation methodology that generates triplets given video-caption pairs, while also expanding the scope of the task to include composed video retrieval (CoVR). To this end, we mine paired videos with a similar caption from a large database, and leverage a large language model to generate the corresponding modification text. Applying this methodology to the extensive WebVid2M collection, we automatically construct our WebVid-CoVR dataset, resulting in 1.6 million triplets. Moreover, we introduce a new benchmark for CoVR with a manually annotated evaluation set, along with baseline results. Our experiments further demonstrate that training a CoVR model on our dataset effectively transfers to CoIR, leading to improved state-of-the-art performance in the zero-shot setup on both the CIRR and FashionIQ benchmarks. Our code, datasets, and models are publicly available at https://imagine.enpc.fr/~ventural/covr.
Zitieren
APA
Ventura, L., Yang, A., Schmid, C., & Varol, G. (2024). CoVR: Learning Composed Video Retrieval from Web Video Captions. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6), 5270-5279. https://doi.org/10.1609/aaai.v38i6.28334
Vancouver
Ventura L, Yang A, Schmid C, Varol G. CoVR: Learning Composed Video Retrieval from Web Video Captions. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(6):5270-5279. doi:10.1609/aaai.v38i6.28334
BibTeX
@article{Ventura2024,
author = {Ventura, Lucas and Yang, Antoine and Schmid, Cordelia and Varol, Gül},
title = {CoVR: Learning Composed Video Retrieval from Web Video Captions},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {6},
pages = {5270--5279},
doi = {10.1609/aaai.v38i6.28334}
}
EUVIMED