EUVIMEDEUVIMED entdecken
Crossref

Enhancing Vision and Language Navigation With Prompt-Based Scene Knowledge

Zhaohuan Zhan, Jinghui Qin, Wei Zhuo, Guang Tan

IEEE Transactions on Circuits and Systems for Video Technology · 2024-10 · 34 (10) · 9745-9756

DOI: 10.1109/tcsvt.2024.3401451 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-10. Verfasst von Zhaohuan Zhan und 3 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 41 Literaturangaben. Gefördert von: Shenzhen Basic Research Fund.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

A challenging task in embodied artificial intelligence is enabling the robot to carry out a navigational task following natural language instruction. In the task, the navigator needs to understand objects, directions, as well as room types, which serve as landmarks for navigation. Although it is easy to encode objects and directions with an external encoder like an object detector, current navigators struggle to encode room type information properly due to the low accuracy offered by existing classifiers. This inadequacy poses confusion that navigators find difficult to overcome. Even humans may sometimes fail to determine the exact type of a room since multiple room types may exist in one panorama. To mitigate this problem, we propose to encode room type information in a prompt manner. Specifically, we first establish multi-modal, learnable prompt pools containing knowledge of room types. By querying the prompt pools, the navigator can obtain room-type prompts of the current view, and incorporate them into the navigator using a prompt-based learning method. Experimental results on the REVERIE, R2R and SOON datasets demonstrate the effectiveness of our approach.

Förderung

  • Shenzhen Basic Research Fund

Zitieren

APA

Zhan, Z., Qin, J., Zhuo, W., & Tan, G. (2024). Enhancing Vision and Language Navigation With Prompt-Based Scene Knowledge. IEEE Transactions on Circuits and Systems for Video Technology, 34(10), 9745-9756. https://doi.org/10.1109/tcsvt.2024.3401451

Vancouver

Zhan Z, Qin J, Zhuo W, Tan G. Enhancing Vision and Language Navigation With Prompt-Based Scene Knowledge. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(10):9745-9756. doi:10.1109/tcsvt.2024.3401451

BibTeX
@article{Zhan2024,
  author = {Zhan, Zhaohuan and Qin, Jinghui and Zhuo, Wei and Tan, Guang},
  title = {Enhancing Vision and Language Navigation With Prompt-Based Scene Knowledge},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2024},
  volume = {34},
  number = {10},
  pages = {9745--9756},
  doi = {10.1109/tcsvt.2024.3401451}
}
Kostenlos starten →