Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention
We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to the target objects and thus makes requests by only specifying high-level end-goals, and (b) the agent is capable of sensing when it is lost and querying an advisor, who is more qualified at the task, to obtain language subgoals to make progress. To model language-based assistance, we develop a general framework termed Imitation Learning with Indirect Intervention (I3L), and propose a solution that is effective on the VNLA task. Empirical results show that this approach significantly improves the success rate of the learning agent over other baselines in both seen and unseen environments. Our code and data are publicly available at https://github.com/debadeepta/vnla .
Code (1)
Tasks
Imitation LearningNavigateVision-based navigation with language-based assistanceVNLASimilar Papers 제목 키워드 기반
Exploring the Use of VLMs for Navigation Assistance for People with Blindness and Low Vision
This paper investigates the potential of vision-language models (VLMs) to assist people with blindness and low vision (pBLV) in navigation tasks. We evaluate state-of-the-art closed-source models, including GPT-4V, GPT-4…
Scene UnderstandingSpatial ReasoningFine-Tuning Vision-Language Models for Visual Navigation Assistance
We address vision-language-driven indoor navigation to assist visually impaired individuals in reaching a target location using images and natural language guidance. Traditional navigation systems are ineffective indoors…
Visual NavigationVisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation
This paper explores the potential of Large Language Models(LLMs) in zero-shot anomaly detection for safe visual navigation. With the assistance of the state-of-the-art real-time open-world object detection model Yolo-Wor…
Anomaly Detectionobject-detectionObject DetectionOpen-vocabulary object detection+5Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
Mobile agents that can leverage help from humans can potentially accomplish more complex tasks than they could entirely on their own. We develop "Help, Anna!" (HANNA), an interactive photo-realistic simulator in which an…
Decision MakingImitation LearningVisual NavigationOrdinal Neural Collapse as a Representation Prior for Visual Navigation
Learning robust navigation policies directly from visual observations remains a fundamental challenge in vision-based robotic navigation. In end-to-end imitation learning approaches, the visual encoder and action decoder…
Visual Navigation