Prompt-guided and multimodal landscape scenicness assessments with vision-language models
PLOS ONE · 30.09.2024 · 19 (9) · e0307083
DOI: 10.1371/journal.pone.0307083 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS ONE, online erschienen am 30.09.2024. Verfasst von Alex Levering und 3 weiteren Autor:innen. Beteiligte Einrichtung: Laboratory of Geo-Information Science and Remote Sensing, Wageningen University, Wageningen, the Netherlands. Hauptthemen laut MEDLINE: Deep Learning. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Recent advances in deep learning and Vision-Language Models (VLM) have enabled efficient transfer to downstream tasks even when limited labelled training data is available, as well as for text to be directly compared to image content. These properties of VLMs enable new opportunities for the annotation and analysis of images. We test the potential of VLMs for landscape scenicness prediction, i.e., the aesthetic quality of a landscape, using zero- and few-shot methods. We experiment with few-shot learning by fine-tuning a single linear layer on a pre-trained VLM representation. We find that a model fitted to just a few hundred samples performs favourably compared to a model trained on hundreds of thousands of examples in a fully supervised way. We also explore the zero-shot prediction potential of contrastive prompting using positive and negative landscape aesthetic concepts. Our results show that this method outperforms a linear probe with few-shot learning when using a small number of samples to tune the prompt configuration. We introduce Landscape Prompt Ensembling (LPE), which is an annotation method for acquiring landscape scenicness ratings through rated text descriptions without needing an image dataset during annotation. We demonstrate that LPE can provide landscape scenicness assessments that are concordant with a dataset of image ratings. The success of zero- and few-shot methods combined with their ability to use text-based annotations highlights the potential for VLMs to provide efficient landscape scenicness assessments with greater flexibility.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Laboratory of Geo-Information Science and Remote Sensing, Wageningen University, Wageningen, the Netherlands.
- Instituut voor Milieuvraagstukken, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands.
- Inria, Université de Montpellier, Montpellier, France.
- McKelvey School of Engineering, Washington University in St. Louis, St. Louis, MO, United States of America.
- Ecole Polytechnique Fédérale de Lausanne, Environmental Computational Science and Earth Observation Laboratory, Sion, Switzerland.
ORCID-Profile: Devis Tuia
Zitieren
APA
Levering, A., Marcos, D., Jacobs, N., & Tuia, D. (2024). Prompt-guided and multimodal landscape scenicness assessments with vision-language models. PLOS ONE, 19(9), e0307083. https://doi.org/10.1371/journal.pone.0307083
Vancouver
Levering A, Marcos D, Jacobs N, Tuia D. Prompt-guided and multimodal landscape scenicness assessments with vision-language models. PLOS ONE. 2024;19(9):e0307083. doi:10.1371/journal.pone.0307083
BibTeX
@article{Levering2024,
author = {Levering, Alex and Marcos, Diego and Jacobs, Nathan and Tuia, Devis},
title = {Prompt-guided and multimodal landscape scenicness assessments with vision-language models},
journal = {PLOS ONE},
year = {2024},
volume = {19},
number = {9},
pages = {e0307083},
doi = {10.1371/journal.pone.0307083}
}
EUVIMED