A Study on Generative Models for Visual Recognition of Unknown Scenes Using a Textual Description
Sensors · 27.10.2023 · 23 (21) · 8757
DOI: 10.3390/s23218757 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 27.10.2023. Verfasst von Jose Martinez-Carranza und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computational Science, Instituto Nacional de Astrofisica, Optica y Electronica (INAOE), Puebla 72840, Mexico. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
In this study, we investigate the application of generative models to assist artificial agents, such as delivery drones or service robots, in visualising unfamiliar destinations solely based on textual descriptions. We explore the use of generative models, such as Stable Diffusion, and embedding representations, such as CLIP and VisualBERT, to compare generated images obtained from textual descriptions of target scenes with images of those scenes. Our research encompasses three key strategies: image generation, text generation, and text enhancement, the latter involving tools such as ChatGPT to create concise textual descriptions for evaluation. The findings of this study contribute to an understanding of the impact of combining generative tools with multi-modal embedding representations to enhance the artificial agent’s ability to recognise unknown scenes. Consequently, we assert that this research holds broad applications, particularly in drone parcel delivery, where an aerial robot can employ text descriptions to identify a destination. Furthermore, this concept can also be applied to other service robots tasked with delivering to unfamiliar locations, relying exclusively on user-provided textual descriptions.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computational Science, Instituto Nacional de Astrofisica, Optica y Electronica (INAOE), Puebla 72840, Mexico.
- Faculty of Computer Science, Benemerita Universidad Autonoma de Puebla (BUAP), Puebla 72570, Mexico.
ORCID-Profile: Jose Martinez-Carranza, Delia Irazú Hernández-Farías, Victoria Eugenia Vazquez-Meza, Leticia Oyuki Rojas-Perez, Aldrich Alfredo Cabrera-Ponce
Zitieren
APA
Martinez-Carranza, J., Hernández-Farías, D. I., Vazquez-Meza, V. E., Rojas-Perez, L. O., & Cabrera-Ponce, A. A. (2023). A Study on Generative Models for Visual Recognition of Unknown Scenes Using a Textual Description. Sensors, 23(21), 8757. https://doi.org/10.3390/s23218757
Vancouver
Martinez-Carranza J, Hernández-Farías DI, Vazquez-Meza VE, Rojas-Perez LO, Cabrera-Ponce AA. A Study on Generative Models for Visual Recognition of Unknown Scenes Using a Textual Description. Sensors. 2023;23(21):8757. doi:10.3390/s23218757
BibTeX
@article{MartinezCarranza2023,
author = {Martinez-Carranza, Jose and Hernández-Farías, Delia Irazú and Vazquez-Meza, Victoria Eugenia and Rojas-Perez, Leticia Oyuki and Cabrera-Ponce, Aldrich Alfredo},
title = {A Study on Generative Models for Visual Recognition of Unknown Scenes Using a Textual Description},
journal = {Sensors},
year = {2023},
volume = {23},
number = {21},
pages = {8757},
doi = {10.3390/s23218757}
}
EUVIMED