Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
Mobile agents that can leverage help from humans can potentially accomplish more complex tasks than they could entirely on their own. We develop "Help, Anna!" (HANNA), an interactive photo-realistic simulator in which an agent fulfills object-finding tasks by requesting and interpreting natural language-and-vision assistance. An agent solving tasks in a HANNA environment can leverage simulated human assistants, called ANNA (Automatic Natural Navigation Assistants), which, upon request, provide natural language and visual instructions to direct the agent towards the goals. To address the HANNA problem, we develop a memory-augmented neural agent that hierarchically models multiple levels of decision-making, and an imitation learning algorithm that teaches the agent to avoid repeating past mistakes while simultaneously predicting its own chances of making future progress. Empirically, our approach is able to ask for help more effectively than competitive baselines and, thus, attains higher task success rate on both previously seen and previously unseen environments. We publicly release code and data at https://github.com/khanhptnk/hanna . A video demo is available at https://youtu.be/18P94aaaLKg .
Code (1)
Tasks
Decision MakingImitation LearningVisual NavigationSimilar Papers 제목 키워드 기반
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
Learning to navigate in a visual environment following natural-language instructions is a challenging task, because the multimodal inputs to the agent are highly variable, and the training data on a new task is often lim…
NavigateSelf-Supervised LearningVision and Language NavigationVisual NavigationAVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments
Recent years have seen embodied visual navigation advance in two distinct directions: (i) in equipping the AI agent to follow natural language instructions, and (ii) in making the navigable world multimodal, e.g., audio-…
AI AgentHierarchical Reinforcement LearningNavigateVisual NavigationUrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectories
Navigating complex urban environments using natural language instructions poses significant challenges for embodied agents, including noisy language instructions, ambiguous spatial references, diverse landmarks, and dyna…
Spatial ReasoningVisual NavigationARIANNA: pAth Recognition for Indoor Assisted NavigatioN with Augmented perception
ARIANNA stands for pAth Recognition for Indoor Assisted Navigation with Augmented perception. It is a flexible and low cost navigation system for vi- sually impaired people. Arianna permits to navigate colored paths pain…
NavigateMultimodal Transformer with Variable-length Memory for Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) is a task that an agent is required to follow a language instruction to navigate to the goal position, which relies on the ongoing interactions with the environment during moving. Rec…
DecoderNavigateVision and Language Navigation