Learning Precise 3D Manipulation from Multiple Uncalibrated Cameras
In this work, we present an effective multi-view approach to closed-loop end-to-end learning of precise manipulation tasks that are 3D in nature. Our method learns to accomplish these tasks using multiple statically placed but uncalibrated RGB camera views without building an explicit 3D representation such as a pointcloud or voxel grid. This multi-camera approach achieves superior task performance on difficult stacking and insertion tasks compared to single-view baselines. Single view robotic agents struggle from occlusion and challenges in estimating relative poses between points of interest. While full 3D scene representations (voxels or pointclouds) are obtainable from registered output of multiple depth sensors, several challenges complicate operating off such explicit 3D representations. These challenges include imperfect camera calibration, poor depth maps due to object properties such as reflective surfaces, and slower inference speeds over 3D representations compared to 2D images. Our use of static but uncalibrated cameras does not require camera-robot or camera-camera calibration making the proposed approach easy to setup and our use of \textit{sensor dropout} during training makes it resilient to the loss of camera-views after deployment.
Code (0)
등록된 구현이 없습니다.
Tasks
Camera CalibrationSimilar Papers 제목 키워드 기반
Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation
Learning to solve precision-based manipulation tasks from visual feedback using Reinforcement Learning (RL) could drastically reduce the engineering efforts required by traditional robot systems. However, performing fine…
Reinforcement Learning (RL)Actron3D: Learning Actionable Neural Functions from Videos for Transferable Robotic Manipulation
We present Actron3D, a framework that enables robots to acquire transferable 6-DoF manipulation skills from just a few monocular, uncalibrated, RGB-only human videos. At its core lies the Neural Affordance Function, a co…
Multi-Person 3D Pose Estimation from Multi-View Uncalibrated Depth Cameras
We tackle the task of multi-view, multi-person 3D human pose estimation from a limited number of uncalibrated depth cameras. Recently, many approaches have been proposed for 3D human pose estimation from multi-view RGB c…
3D Human Pose Estimation3D Pose Estimation3D ReconstructionCamera Pose Estimation+2Automatic Construction of Real-World Datasets for 3D Object Localization using Two Cameras
Unlike classification, position labels cannot be assigned manually by humans. For this reason, generating supervision for precise object localization is a hard task. This paper details a method to create large datasets f…
General ClassificationObjectObject LocalizationPositionConstrained Bundle Adjustment for Structure From Motion Using Uncalibrated Multi-Camera Systems
Structure from motion using uncalibrated multi-camera systems is a challenging task. This paper proposes a bundle adjustment solution that implements a baseline constraint respecting that these cameras are static to each…