Action-Aware Visual-Textual Alignment for Long-Instruction Vision-and-Language Navigation
ACM Transactions on Multimedia Computing, Communications, and Applications · 30.09.2025 · 21 (9) · 1-22
DOI: 10.1145/3748656 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 11.09.2025. Verfasst von Bowen Huang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 74 Literaturangaben. Gefördert von: Natural Science Foundation of Shandong Province, China.
Abstract (Originalsprache)
Traditional Vision-and-Language Navigation (VLN) requires an agent to navigate to a target location solely based on visual observations, guided by natural language instructions. Compared to this task, long-instruction VLN involves longer instructions, extended trajectories, and the need to consider more contextual information for global path planning. As a result, it is more challenging and requires accurately aligning the instructions with the agent’s current visual observations, which is accompanied by two significant issues. Firstly, there is a misalignment between actions. The visual observations of the agent at each step lack explicit action-related details, while the instructions contain action-oriented words. Secondly, there is a misalignment between global instructions and local visual observations. The instructions describe the entire navigation trajectory, whereas the agent’s visual observations only provide localized information about a specific position along the trajectory. To address these issues, this article introduces the Action-Perception Alignment Framework (APAF). In this framework, we first design the Action-Contextual Encoding Module (ACEM), which enriches the agent’s visual perception by encoding potential actions with relative heading and elevation angles. We then propose the Dynamic Instruction Weighting Module (DIWM), which adjusts the importance of instruction words based on the agent’s current visual observations, emphasizing those words most relevant to the agent’s visual observations. Our approach significantly outperforms existing methods, achieving state-of-the-art results with improvements of 8.5% and 4.0% in Success Rate (SR) on the long-instruction R4R and RxR datasets, respectively.
Förderung
- Natural Science Foundation of Shandong Province, China
Zitieren
APA
Huang, B., Zheng, Y., Lan, C., Sui, D., Zhao, X., Zhang, X., Xiao, M., & Yu, D. (2025). Action-Aware Visual-Textual Alignment for Long-Instruction Vision-and-Language Navigation. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(9), 1-22. https://doi.org/10.1145/3748656
Vancouver
Huang B, Zheng Y, Lan C, Sui D, Zhao X, Zhang X, et al. Action-Aware Visual-Textual Alignment for Long-Instruction Vision-and-Language Navigation. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(9):1-22. doi:10.1145/3748656
BibTeX
@article{Huang2025,
author = {Huang, Bowen and Zheng, Yanwei and Lan, Chuanlin and Sui, Dongchen and Zhao, Xinpeng and Zhang, Xiao and Xiao, Mengbai and Yu, Dongxiao},
title = {Action-Aware Visual-Textual Alignment for Long-Instruction Vision-and-Language Navigation},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2025},
volume = {21},
number = {9},
pages = {1--22},
doi = {10.1145/3748656}
}
EUVIMED