Towards Embodied Scene Description
Embodiment is an important characteristic for all intelligent agents (creatures and robots), while existing scene description tasks mainly focus on analyzing images passively and the semantic understanding of the scenario is separated from the interaction between the agent and the environment. In this work, we propose the Embodied Scene Description, which exploits the embodiment ability of the agent to find an optimal viewpoint in its environment for scene description tasks. A learning framework with the paradigms of imitation learning and reinforcement learning is established to teach the intelligent agent to generate corresponding sensorimotor activities. The proposed framework is tested on both the AI2Thor dataset and a real world robotic platform demonstrating the effectiveness and extendability of the developed method.
Code (0)
등록된 구현이 없습니다.
Tasks
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding
3D occupancy prediction provides a comprehensive description of the surrounding scenes and has become an essential task for 3D perception. Most existing methods focus on offline perception from one or a few views and can…
PredictionScene UnderstandingEmbodied Agents for Efficient Exploration and Smart Scene Description
The development of embodied agents that can communicate with humans in natural language has gained increasing interest over the last years, as it facilitates the diffusion of robotic platforms in human-populated environm…
Efficient ExplorationImage CaptioningVisual NavigationVision-Language Embodiment for Monocular Depth Estimation
Depth estimation is a core problem in robotic perception and vision tasks, but 3D reconstruction from a single image presents inherent uncertainties. Current depth estimation models primarily rely on inter-image rela…
3D ReconstructionDepth EstimationMonocular Depth EstimationScene Understanding"Where am I?" Scene Retrieval with Language
Natural language interfaces to embodied AI are becoming more ubiquitous in our daily lives. This opens up further opportunities for language-based interaction with embodied agents, such as a user verbally instructing an …
RetrievalExplore and Tell: Embodied Visual Captioning in 3D Environments
While current visual captioning models have achieved impressive performance, they often assume that the image is well-captured and provides a complete view of the scene. In real-world scenarios, however, a single image m…
Image CaptioningNavigateScene Understanding