CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (4) · 3765-3773
DOI: 10.1609/aaai.v38i4.28167 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Xiulong Liu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref).
Abstract (Originalsprache)
Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy. In this paper, we present CAVEN, a Conversation-based Audio-Visual Embodied Navigation framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal. Specifically, CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that implicitly learns the uncertainty in the audio-based navigation policy to decide when and how the agent may interact with the oracle. Our CAVEN agent can engage in fully-bidirectional natural language conversations by producing relevant questions and interpret free-form, potentially noisy responses from the oracle based on the audio-visual context. To enable such a capability, CAVEN is equipped with: i) a trajectory forecasting network that is grounded in audio-visual cues to produce a potential trajectory to the estimated goal, and (ii) a natural language based question generation and reasoning network to pose an interactive question to the oracle or interpret the oracle's response to produce navigation instructions. To train the interactive modules, we present a large scale dataset: AVN-Instruct, based on the Landmark-RxR dataset. To substantiate the usefulness of conversations, we present experiments on the benchmark audio-goal task using the SoundSpaces simulator under various noisy settings. Our results reveal that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods that use only uni-directional interaction.
Zitieren
APA
Liu, X., Paul, S., Chatterjee, M., & Cherian, A. (2024). CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments. Proceedings of the AAAI Conference on Artificial Intelligence, 38(4), 3765-3773. https://doi.org/10.1609/aaai.v38i4.28167
Vancouver
Liu X, Paul S, Chatterjee M, Cherian A. CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(4):3765-3773. doi:10.1609/aaai.v38i4.28167
BibTeX
@article{Liu2024,
author = {Liu, Xiulong and Paul, Sudipta and Chatterjee, Moitreya and Cherian, Anoop},
title = {CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {4},
pages = {3765--3773},
doi = {10.1609/aaai.v38i4.28167}
}
EUVIMED