An Exploration of Embodied Visual Exploration
Embodied computer vision considers perception for robots in novel, unstructured environments. Of particular importance is the embodied visual exploration problem: how might a robot equipped with a camera scope out a new environment? Despite the progress thus far, many basic questions pertinent to this problem remain unanswered: (i) What does it mean for an agent to explore its environment well? (ii) Which methods work well, and under which assumptions and environmental settings? (iii) Where do current approaches fall short, and where might future work seek to improve? Seeking answers to these questions, we first present a taxonomy for existing visual exploration algorithms and create a standard framework for benchmarking them. We then perform a thorough empirical study of the four state-of-the-art paradigms using the proposed framework with two photorealistic simulated 3D environments, a state-of-the-art exploration architecture, and diverse evaluation metrics. Our experimental results offer insights and suggest new performance metrics and baselines for future work in visual exploration. Code, models and data are publicly available: https://github.com/facebookresearch/exploring_exploration
Code (1)
Tasks
BenchmarkingSimilar Papers 제목 키워드 기반
Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answering
Embodied Question Answering (EQA) is a challenging task in embodied intelligence that requires agents to dynamically explore 3D environments, actively gather visual information, and perform multi-step reasoning to answer…
Embodied Question AnsweringQuestion AnsweringUnderstanding Representations Pretrained with Auxiliary Losses for Embodied Agent Planning
Pretrained representations from large-scale vision models have boosted the performance of downstream embodied policy learning. We look to understand whether additional self-supervised pretraining on exploration trajector…
Imitation LearningReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replay
Embodied exploration is a target-driven process that requires embodied agents to possess fine-grained perception and knowledge-enhanced decision making. While recent attempts leverage MLLMs for exploration due to their s…
Reinforcement LearningDecision MakingObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Exploration
Embodied reasoning and exploration are increasingly considered crucial abilities for robots operating in complex and unfamiliar environments. To accomplish tasks in such settings, an agent must identify and acquire the i…
HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering
Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on tex…
Semantic SimilarityQuestion AnsweringVisual Reasoning