Enhancing Vision and Language Navigation With Prompt-Based Scene Knowledge
IEEE Transactions on Circuits and Systems for Video Technology · 2024-10 · 34 (10) · 9745-9756
DOI: 10.1109/tcsvt.2024.3401451 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-10. Verfasst von Zhaohuan Zhan und 3 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 41 Literaturangaben. Gefördert von: Shenzhen Basic Research Fund.
Abstract (Originalsprache)
A challenging task in embodied artificial intelligence is enabling the robot to carry out a navigational task following natural language instruction. In the task, the navigator needs to understand objects, directions, as well as room types, which serve as landmarks for navigation. Although it is easy to encode objects and directions with an external encoder like an object detector, current navigators struggle to encode room type information properly due to the low accuracy offered by existing classifiers. This inadequacy poses confusion that navigators find difficult to overcome. Even humans may sometimes fail to determine the exact type of a room since multiple room types may exist in one panorama. To mitigate this problem, we propose to encode room type information in a prompt manner. Specifically, we first establish multi-modal, learnable prompt pools containing knowledge of room types. By querying the prompt pools, the navigator can obtain room-type prompts of the current view, and incorporate them into the navigator using a prompt-based learning method. Experimental results on the REVERIE, R2R and SOON datasets demonstrate the effectiveness of our approach.
Förderung
- Shenzhen Basic Research Fund
Zitieren
APA
Zhan, Z., Qin, J., Zhuo, W., & Tan, G. (2024). Enhancing Vision and Language Navigation With Prompt-Based Scene Knowledge. IEEE Transactions on Circuits and Systems for Video Technology, 34(10), 9745-9756. https://doi.org/10.1109/tcsvt.2024.3401451
Vancouver
Zhan Z, Qin J, Zhuo W, Tan G. Enhancing Vision and Language Navigation With Prompt-Based Scene Knowledge. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(10):9745-9756. doi:10.1109/tcsvt.2024.3401451
BibTeX
@article{Zhan2024,
author = {Zhan, Zhaohuan and Qin, Jinghui and Zhuo, Wei and Tan, Guang},
title = {Enhancing Vision and Language Navigation With Prompt-Based Scene Knowledge},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2024},
volume = {34},
number = {10},
pages = {9745--9756},
doi = {10.1109/tcsvt.2024.3401451}
}
EUVIMED