Cross-modal Map Learning for Vision and Language Navigation
We consider the problem of Vision-and-Language Navigation (VLN). The majority of current methods for VLN are trained end-to-end using either unstructured memory such as LSTM, or using cross-modal attention over the egocentric observations of the agent. In contrast to other works, our key insight is that the association between language and vision is stronger when it occurs in explicit spatial representations. In this work, we propose a cross-modal map learning model for vision-and-language navigation that first learns to predict the top-down semantics on an egocentric map for both observed and unobserved regions, and then predicts a path towards the goal as a set of waypoints. In both cases, the prediction is informed by the language through cross-modal attention mechanisms. We experimentally test the basic hypothesis that language-driven navigation can be solved given a map, and then show competitive results on the full VLN-CE benchmark.
Code (1)
Tasks
Vision and Language NavigationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Vision-Dialog Navigation by Exploring Cross-modal Memory
Vision-dialog navigation posed as a new holy-grail task in vision-language disciplinary targets at learning an agent endowed with the capability of constant conversation for help with natural language and navigating acco…
Decision MakingDiagnosing Vision-and-Language Navigation: What Really Matters
Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model arc…
DiagnosticObjectVision and Language NavigationPlatonicNav: Unveiling Semantic Correspondence in Navigation with Platonic Topological Maps
Embodied visual navigation, where an agent perceives a complex environment and acts to reach a goal from raw sensory input, underpins a wide range of applications such as household service robotics, assistive robotics, a…
Semantic correspondenceVisual NavigationDELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learning
Vision-and-Language navigation (VLN) requires an agent to navigate in unseen environment by following natural language instruction. For task completion, the agent needs to align and integrate various navigation modalitie…
Contrastive LearningDecision MakingNavigateVision and Language NavigationLandmark-RxR: Solving Vision-and-Language Navigation with Fine-Grained Alignment Supervision
In Vision-and-Language Navigation (VLN) task, an agent is asked to navigate inside 3D indoor environments following given instructions. Cross-modal alignment is one of the most critical challenges in VLN because the pred…
cross-modal alignmentNavigateVision and Language Navigation