Embodied Visual Recognition

2019-04-09 · Jianwei Yang, Zhile Ren, Mingze Xu, Xinlei Chen, David Crandall, Devi Parikh, Dhruv Batra

Passive visual systems typically fail to recognize objects in the amodal setting where they are heavily occluded. In contrast, humans and other embodied agents have the ability to move in the environment, and actively control the viewing angle to better understand object shapes and semantics. In this work, we introduce the task of Embodied Visual Recognition (EVR): An agent is instantiated in a 3D environment close to an occluded target object, and is free to move in the environment to perform object classification, amodal object localization, and amodal object segmentation. To address this, we develop a new model called Embodied Mask R-CNN, for agents to learn to move strategically to improve their visual recognition abilities. We conduct experiments using the House3D environment. Experimental results show that: 1) agents with embodiment (movement) achieve better visual recognition performance than passive ones; 2) in order to improve visual recognition abilities, agents can learn strategical moving paths that are different from shortest paths.

📄 PDF Abstract BibTeX arXiv:1904.04404

Code (0)

등록된 구현이 없습니다.

Tasks

ObjectObject Localization Semantic Segmentation

Methods 이 논문이 사용한 방법론

RPN A Region Proposal Network, or RPN, is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

RoIAlign Region of Interest Align, or RoIAlign, is an operation for extracting a small feature map from each RoI in detection and segmentation based tasks. It removes the harsh…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Mask R-CNN Mask R-CNN extends Faster R-CNN to solve instance segmentation tasks. It achieves this by adding a branch for predicting an…

Embodied Amodal Recognition: Learning to Move to Perceive Objects

2019-10-01 · ICCV 2019 10 · Jianwei Yang, Zhile Ren, Mingze Xu, Xinlei Chen 외

Object Object Localization Semantic Segmentation

TrackVLA: Embodied Visual Tracking in the Wild

2025-05-29 · Shaoan Wang, Jiazhao Zhang, Minghan Li, Jiahang Liu 외

Embodied visual tracking is a fundamental skill in Embodied AI, enabling an agent to follow a specific target in dynamic environments using only egocentric vision. This task is inherently challenging as it requires both …

Language Modeling Language Modelling Object Recognition Trajectory Planning+2

RoboTHOR: An Open Simulation-to-Real Embodied AI Platform

2020-04-14 · CVPR 2020 6 · Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi 외

Visual recognition ecosystems (e.g. ImageNet, Pascal, COCO) have undeniably played a prevailing role in the evolution of modern computer vision. We argue that interactive and embodied visual AI has reached a stage of dev…

Multimodal Speech Recognition for Language-Guided Embodied Agents

2023-02-27 · Allen Chang, Xiaoyuan Zhu, Aarav Monga, Seoho Ahn 외

Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions. While Automatic Speech Recognition (ASR) models can bridge the input gap, e…

Automatic Speech Recognition Automatic Speech Recognition (ASR)speech-recognition Speech Recognition

SeekNet: Improved Human Instance Segmentation and Tracking via Reinforcement Learning Based Optimized Robot Relocation

2020-11-17 · Venkatraman Narayanan, Bala Murali Manoghar, Rama Prashanth RV, Phu Pham 외

Amodal recognition is the ability of the system to detect occluded objects. Most SOTA Visual Recognition systems lack the ability to perform amodal recognition. Few studies have achieved amodal recognition through passiv…

Human Detection Human Instance Segmentation Instance Segmentation reinforcement-learning+2