Learning to compose 6-DoF omnidirectional videos using multi-sphere images
Omnidirectional video is an essential component of Virtual Reality. Although various methods have been proposed to generate content that can be viewed with six degrees of freedom (6-DoF), existing systems usually involve complex depth estimation, image in-painting or stitching pre-processing. In this paper, we propose a system that uses a 3D ConvNet to generate a multi-sphere images (MSI) representation that can be experienced in 6-DoF VR. The system utilizes conventional omnidirectional VR camera footage directly without the need for a depth map or segmentation mask, thereby significantly simplifying the overall complexity of the 6-DoF omnidirectional video composition. By using a newly designed weighted sphere sweep volume (WSSV) fusing technique, our approach is compatible with most panoramic VR camera setups. A ground truth generation approach for high-quality artifact-free 6-DoF contents is proposed and can be used by the research and development community for 6-DoF content generation.
Code (0)
등록된 구현이 없습니다.
Tasks
Depth EstimationSimilar Papers 제목 키워드 기반
Real-Time Sphere Sweeping Stereo From Multiview Fisheye Images
A set of cameras with fisheye lenses have been used to capture a wide field of view. The traditional scan-line stereo algorithms based on epipolar geometry are directly inapplicable to this non-pinhole camera setup d…
GPUOSLO: On-the-Sphere Learning for Omnidirectional images and its application to 360-degree image compression
State-of-the-art 2D image compression schemes rely on the power of convolutional neural networks (CNNs). Although CNNs offer promising perspectives for 2D image compression, extending such models to omnidirectional image…
Image CompressionSphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images
Omnidirectional cameras offer great benefits over classical cameras wherever a wide field of view is essential, such as in virtual reality applications or in autonomous robots. Unfortunately, standard convolutional neura…
General Classificationimage-classificationImage Classificationobject-detection+1ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian Splattings
Omnidirectional (or 360-degree) images are increasingly being used for 3D applications since they allow the rendering of an entire scene with a single image. Existing works based on neural radiance fields demonstrate suc…
3D Reconstruction3D Scene ReconstructionMathematical ProofsNeRFOmniMVS: End-to-End Learning for Omnidirectional Stereo Matching
In this paper, we propose a novel end-to-end deep neural network model for omnidirectional depth estimation from a wide-baseline multi-view stereo setup. The images captured with ultra wide field-of-view (FOV) cameras on…
DecoderDepth EstimationStereo MatchingStereo Matching Hand