Crowdsampling the Plenoptic Function
Many popular tourist landmarks are captured in a multitude of online, public photos. These photos represent a sparse and unstructured sampling of the plenoptic function for a particular scene. In this paper,we present a new approach to novel view synthesis under time-varying illumination from such data. Our approach builds on the recent multi-plane image (MPI) format for representing local light fields under fixed viewing conditions. We introduce a new DeepMPI representation, motivated by observations on the sparsity structure of the plenoptic function, that allows for real-time synthesis of photorealistic views that are continuous in both space and across changes in lighting. Our method can synthesize the same compelling parallax and view-dependent effects as previous MPI methods, while simultaneously interpolating along changes in reflectance and illumination with time. We show how to learn a model of these effects in an unsupervised way from an unstructured collection of photos without temporal registration, demonstrating significant improvements over recent work in neural rendering. More information can be found crowdsampling.io.
Code (1)
Tasks
Neural RenderingNovel View SynthesisSimilar Papers 제목 키워드 기반
Neural Plenoptic Sampling: Capture Light-field from Imaginary Eyes
The Plenoptic function describes the light rays observed from any given position in every viewing direction. It is often parameterized as a 5-D function $L(x, y, z, \theta, \phi)$ for a static scene. Capturing all the p…
Novel View SynthesisPositionReflection Separation and Deblurring of Plenoptic Images
In this paper, we address the problem of reflection removal and deblurring from a single image captured by a plenoptic camera. We develop a two-stage approach to recover the scene depth and high resolution textures of th…
DeblurringDepth EstimationReflection RemovalBlur Aware Calibration of Multi-Focus Plenoptic Camera
This paper presents a novel calibration algorithm for Multi-Focus Plenoptic Cameras (MFPCs) using raw images only. The design of such cameras is usually complex and relies on precise placement of optic elements. Several …
Blur Aware Calibration of Multi-Focus Plenoptic Camera
This paper presents a novel calibration algorithm for Multi-Focus Plenoptic Cameras (MFPCs) using raw images only. The design of such cameras is usually complex and relies on precise placement of optic elements. Several …
PRISM-VO: Scale-Aware Visual Odometry Using Photometric Plenoptic Bundle Adjustment
We introduce PRISM-VO, a novel pure optimization-based sparse photometric visual odometry framework for focused plenoptic cameras. The core of PRISM-VO is a novel photometric plenoptic bundle adjustment which jointly opt…
Visual Odometry