RAFT-3D: Scene Flow using Rigid-Motion Embeddings
We address the problem of scene flow: given a pair of stereo or RGB-D video frames, estimate pixelwise 3D motion. We introduce RAFT-3D, a new deep architecture for scene flow. RAFT-3D is based on the RAFT model developed for optical flow but iteratively updates a dense field of pixelwise SE3 motion instead of 2D motion. A key innovation of RAFT-3D is rigid-motion embeddings, which represent a soft grouping of pixels into rigid objects. Integral to rigid-motion embeddings is Dense-SE3, a differentiable layer that enforces geometric consistency of the embeddings. Experiments show that RAFT-3D achieves state-of-the-art performance. On FlyingThings3D, under the two-view evaluation, we improved the best published accuracy (d < 0.05) from 34.3% to 83.7%. On KITTI, we achieve an error of 5.77, outperforming the best published method (6.31), despite using no object instance supervision. Code is available at https://github.com/princeton-vl/RAFT-3D.
Code (1)
Tasks
Optical Flow EstimationScene Flow EstimationSimilar Papers 제목 키워드 기반
Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion
Most of the current scene flow methods choose to model scene flow as a per point translation vector without differentiating between static and dynamic components of 3D motion. In this work we present an alternative metho…
Self-Supervised LearningTranslationTARS: Traffic-Aware Radar Scene Flow Estimation
Scene flow provides crucial motion information for autonomous driving. Recent LiDAR scene flow models utilize the rigid-motion assumption at the instance level, assuming objects are rigid bodies. However, these instance-…
Autonomous Drivingobject-detectionObject DetectionScene Flow Estimation+1RigidFlow: Self-Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior
In this work, we focus on scene flow learning on point clouds in a self-supervised manner. A real-world scene can be well modeled as a collection of rigidly moving parts, therefore its scene flow can be represented a…
Motion EstimationSelf-Supervised LearningLearning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems. In real world applications, a dynamic scene is commonly captured by a moving camera (i.e., pa…
Optical Flow EstimationScene Flow EstimationScene UnderstandingMultiframe Scene Flow with Piecewise Rigid Motion
We introduce a novel multiframe scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences. In contrast to the competing methods, we take …
Scene Flow Estimation