WildCLIP: Scene and Animal Attribute Retrieval from Camera Trap Data with Domain-Adapted Vision-Language Models
International Journal of Computer Vision · 2024-09 · 132 (9) · 3770-3786
DOI: 10.1007/s11263-024-02026-6 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 24.04.2024. Verfasst von Valentin Gabeff und 3 weiteren Autor:innen. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: EPFL Lausanne.
Abstract (Originalsprache)
Wildlife observation with camera traps has great potential for ethology and ecology, as it gathers data non-invasively in an automated way. However, camera traps produce large amounts of uncurated data, which is time-consuming to annotate. Existing methods to label these data automatically commonly use a fixed pre-defined set of distinctive classes and require many labeled examples per class to be trained. Moreover, the attributes of interest are sometimes rare and difficult to find in large data collections. Large pretrained vision-language models, such as contrastive language image pretraining (CLIP), offer great promises to facilitate the annotation process of camera-trap data. Images can be described with greater detail, the set of classes is not fixed and can be extensible on demand and pretrained models can help to retrieve rare samples. In this work, we explore the potential of CLIP to retrieve images according to environmental and ecological attributes. We create WildCLIP by fine-tuning CLIP on wildlife camera-trap images and to further increase its flexibility, we add an adapter module to better expand to novel attributes in a few-shot manner. We quantify WildCLIP’s performance and show that it can retrieve novel attributes in the Snapshot Serengeti dataset. Our findings outline new opportunities to facilitate annotation processes with complex and multi-attribute captions. The code is available at https://github.com/amathislab/wildclip .
Förderung
- EPFL Lausanne
Zitieren
APA
Gabeff, V., Rußwurm, M., Tuia, D., & Mathis, A. (2024). WildCLIP: Scene and Animal Attribute Retrieval from Camera Trap Data with Domain-Adapted Vision-Language Models. International Journal of Computer Vision, 132(9), 3770-3786. https://doi.org/10.1007/s11263-024-02026-6
Vancouver
Gabeff V, Rußwurm M, Tuia D, Mathis A. WildCLIP: Scene and Animal Attribute Retrieval from Camera Trap Data with Domain-Adapted Vision-Language Models. International Journal of Computer Vision. 2024;132(9):3770-3786. doi:10.1007/s11263-024-02026-6
BibTeX
@article{Gabeff2024,
author = {Gabeff, Valentin and Rußwurm, Marc and Tuia, Devis and Mathis, Alexander},
title = {WildCLIP: Scene and Animal Attribute Retrieval from Camera Trap Data with Domain-Adapted Vision-Language Models},
journal = {International Journal of Computer Vision},
year = {2024},
volume = {132},
number = {9},
pages = {3770--3786},
doi = {10.1007/s11263-024-02026-6}
}
EUVIMED