Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments. This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal. In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress. We test our self-monitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components. Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set). Code is available at https://github.com/chihyaoma/selfmonitoring-agent .
Code (2)
Tasks
Natural Language Visual GroundingVision and Language NavigationVision-Language NavigationVisual NavigationSimilar Papers 제목 키워드 기반
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
Vision-Language Navigation (VLN) is a task where agents learn to navigate following natural language instructions. The key to this task is to perceive both the visual scene and natural language sequentially. Conventional…
NavigateVision-Language NavigationRealistic PointGoal Navigation via Auxiliary Losses and Information Bottleneck
We propose a novel architecture and training paradigm for training realistic PointGoal Navigation -- navigating to a target coordinate in an unseen environment under actuation and sensor noise without access to ground-tr…
PointGoal NavigationACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI
Embodied AI is moving from isolated perception or action modules toward physical agents that understand, plan under goals, act through robot bodies, monitor progress, and improve from experience. Existing systems address…
Spatial ReasoningDecision MakingAuxiliary Tasks Speed Up Learning PointGoal Navigation
PointGoal Navigation is an embodied task that requires agents to navigate to a specified point in an unseen environment. Wijmans et al. showed that this task is solvable but their method is computationally prohibitive, r…
GPUNavigatePointGoal NavigationA Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings
In this work we explore an auxiliary loss useful for reinforcement learning in environments where strong performing agents are required to be able to navigate a spatial environment. The auxiliary loss proposed is to mini…
NavigateSpatial Reasoning