Active Perception and Representation for Robotic Manipulation

2020-03-15 · Youssef Zaky, Gaurav Paruthi, Bryan Tripp, James Bergstra

The vast majority of visual animals actively control their eyes, heads, and/or bodies to direct their gaze toward different parts of their environment. In contrast, recent applications of reinforcement learning in robotic manipulation employ cameras as passive sensors. These are carefully placed to view a scene from a fixed pose. Active perception allows animals to gather the most relevant information about the world and focus their computational resources where needed. It also enables them to view objects from different distances and viewpoints, providing a rich visual experience from which to learn abstract representations of the environment. Inspired by the primate visual-motor system, we present a framework that leverages the benefits of active perception to accomplish manipulation tasks. Our agent uses viewpoint changes to localize objects, to learn state representations in a self-supervised manner, and to perform goal-directed actions. We apply our model to a simulated grasping task with a 6-DoF action space. Compared to its passive, fixed-camera counterpart, the active model achieves 8% better performance in targeted grasping. Compared to vanilla deep Q-learning algorithms, our model is at least four times more sample-efficient, highlighting the benefits of both active perception and representation learning.

📄 PDF Abstract BibTeX arXiv:2003.06734

Code (0)

등록된 구현이 없습니다.

Tasks

Q-LearningReinforcement Learning Representation Learning

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint

2026-01-20 · Deyun Qin, Zezhi Liu, Hanqian Luo, Xiao Liang 외 arxiv

Active perception in vision-based robotic manipulation aims to move the camera toward more informative observation viewpoints, thereby providing high-quality perceptual inputs for downstream tasks. Most existing active p…

Robotic Grasping

Object SLAM-Based Active Mapping and Robotic Grasping

2020-12-03 · Yanmin Wu, Yunzhou Zhang, Delong Zhu, Xin Chen 외

This paper presents the first active object mapping framework for complex robotic manipulation and autonomous perception tasks. The framework is built on an object SLAM system integrated with a simultaneous multi-object …

Object Object SLAM Pose Estimation Robotic Grasping

SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Robotics

2026-03-12 · Mengzhen Liu, Enshen Zhou, Cheng Chi, Yi Han 외 arxiv

Active perception and manipulation are crucial for robots to interact with complex scenes. Existing methods struggle to unify semantic-driven active perception with robust, viewpoint-invariant execution. We propose SaPaV…

ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation

2026-01-13 · Zhenyang Liu, Yongchong Gu, Yikai Wang, Xiangyang Xue 외 arxiv

Recent advances in robot manipulation have leveraged pre-trained vision-language models (VLMs) and explored integrating 3D spatial signals into these models for effective action prediction, giving rise to the promising v…

Robot Manipulation

ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration

2026-04-09 · Yanwen Zou, Chenyang Shi, Wenye Yu, Han Xue 외 arxiv

Large-scale real-world robot data collection is a prerequisite for bringing robots into everyday deployment. However, existing pipelines often rely on specialized handheld devices to bridge the embodiment gap, which not …