Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality
Lighting understanding plays an important role in virtual object composition, including mobile augmented reality (AR) applications. Prior work often targets recovering lighting from the physical environment to support photorealistic AR rendering. Because the common workflow is to use a back-facing camera to capture the physical world for overlaying virtual objects, we refer to this usage pattern as back-facing AR. However, existing methods often fall short in supporting emerging front-facing mobile AR applications, e.g., virtual try-on where a user leverages a front-facing camera to explore the effect of various products (e.g., glasses or hats) of different styles. This lack of support can be attributed to the unique challenges of obtaining 360$^\circ$ HDR environment maps, an ideal format of lighting representation, from the front-facing camera and existing techniques. In this paper, we propose to leverage dual-camera streaming to generate a high-quality environment map by combining multi-view lighting reconstruction and parametric directional lighting estimation. Our preliminary results show improved rendering quality using a dual-camera setup for front-facing AR compared to a commercial solution.
Code (0)
등록된 구현이 없습니다.
Tasks
Lighting EstimationVirtual Try-onSimilar Papers 제목 키워드 기반
SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
We introduce SceneNet RGB-D, expanding the previous work of SceneNet to enable large scale photorealistic rendering of indoor scene trajectories. It provides pixel-perfect ground truth for scene understanding problems su…
3D ReconstructionCamera Pose EstimationDepth EstimationInstance Segmentation+6Total Variation-Based Dense Depth from Multi-Camera Array
Multi-Camera arrays are increasingly employed in both consumer and industrial applications, and various passive techniques are documented to estimate depth from such camera arrays. Current depth estimation methods provid…
Depth EstimationSIDOD: A Synthetic Image Dataset for 3D Object Pose Recognition with Distractors
We present a new, publicly-available image dataset generated by the NVIDIA Deep Learning Data Synthesizer intended for use in object detection, pose estimation, and tracking applications. This dataset contains 144k stere…
Objectobject-detectionObject DetectionPose EstimationWaterSplat-SLAM: Photorealistic Monocular SLAM in Underwater Environment
Underwater monocular SLAM is a challenging problem with applications from autonomous underwater vehicles to marine archaeology. However, existing underwater SLAM methods struggle to produce maps with high-fidelity render…
3D ReconstructionDepth EstimationPose EstimationIRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images
Inverse rendering seeks to recover 3D geometry, surface material, and lighting from captured images, enabling advanced applications such as novel-view synthesis, relighting, and virtual object insertion. However, most ex…
3D geometry3D ReconstructionInverse RenderingNovel View Synthesis