Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression
The Remote Embodied Referring Expression (REVERIE) is a recently raised task that requires an agent to navigate to and localise a referred remote object according to a high-level language instruction. Different from related VLN tasks, the key to REVERIE is to conduct goal-oriented exploration instead of strict instruction-following, due to the lack of step-by-step navigation guidance. In this paper, we propose a novel Cross-modality Knowledge Reasoning (CKR) model to address the unique challenges of this task. The CKR, based on a transformer-architecture, learns to generate scene memory tokens and utilise these informative history clues for exploration. Particularly, a Room-and-Object Aware Attention (ROAA) mechanism is devised to explicitly perceive the room- and object-type information from both linguistic and visual observations. Moreover, through incorporating commonsense knowledge, we propose a Knowledge-enabled Entity Relationship Reasoning (KERR) module to learn the internal-external correlations among room- and object-entities for agent to make proper action at each viewpoint. Evaluation on REVERIE benchmark demonstrates the superiority of the CKR model, which significantly boosts SPL and REVERIE-success rate by 64.67% and 46.05%, respectively. Code is available at: https://github.com/alloldman/CKR.
Code (1)
Tasks
Instruction FollowingNavigateObjectReferring ExpressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Layout-aware Dreamer for Embodied Referring Expression Grounding
In this work, we study the problem of Embodied Referring Expression Grounding, where an agent needs to navigate in a previously unseen environment and localize a remote object described by a concise high-level natural la…
Common Sense ReasoningNavigateReferring ExpressionMMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation
The fusion of Large Language Models with vision models is pioneering new possibilities in user-interactive vision-language tasks. A notable application is reasoning segmentation, where models generate pixel-level segment…
ObjectReasoning SegmentationSegmentationCAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models
Existing 3D scene-grounded Large Language Models (3D-LLMs) focus on answering questions grounded in simplified single-room 3D scenes, lacking the ability to reason over real-world household environments containing multip…
Graph Neural NetworkScene UnderstandingMarch in Chat: Interactive Prompting for Remote Embodied Referring Expression
Many Vision-and-Language Navigation (VLN) tasks have been proposed in recent years, from room-based to object-based and indoor to outdoor. The REVERIE (Remote Embodied Referring Expression) is interesting since it only p…
Referring ExpressionVision and Language NavigationThink and Answer ME: Benchmarking and Exploring Multi-Entity Reasoning Grounding in Remote Sensing
Recent advances in reasoning language models and reinforcement learning with verifiable rewards have significantly enhanced multi-step reasoning capabilities. This progress motivates the extension of reasoning paradigms …
Reinforcement LearningVisual Grounding